Building a Kubernetes Cluster With Raspberry Pis and containerd

I’ve found a lot of guides on how to get Kubernetes set up on Raspberry Pis (Raspberry Pi 4 specifically), but I haven’t found any recently that go through the process using containerd instead of Docker. I recently worked through this process and thought I’d share my experience.

Before we get started, I wanted to talk about my current setup a little bit as it’s not 100% the latest/stock.

  • Firstly, I’m running Ubuntu 20.04 Server on each of my Raspberry Pi nodes. I’m currently holding off on upgrading to 22.04 as when we did that at work, it broke our Kubernetes cluster. I currently don’t want to fight with that at home. Hopefully, the build instructions will work on 22.04 as a fresh install, but YMMV.
  • Secondly, I’ve gone through the process of booting my Raspberry Pis from SSD instead of microSD cards — I’ll make a post about how I did that in the future. This shouldn’t change the build procedure, but I wanted to call it out.

Preparation and prerequisites

What you’ll need:

  • Raspberry Pis – ideally you’d want at least three. This way you can have one control-plane and two workers.
  • Storage – No matter how you’re booting your pis, you’ll want to make sure you’ve got a decent amount of free space for ephemeral storage (this is the space containers will use as “local” storage unless you’re mounting persistent storage somewhere else, like via NFS). For SD Cards, 64GB or 128GB should suffice. For SSDs, you can say the same, but they’re so cheap now I’ve been going with 250GB SSDs just because.
  • Networking – you’ll want these Pis to be able to talk to each other and the internet (you need to get your container images from somewhere).
  • (Optional) If you’ve got a PoE switch and PoE hats for your Pis, I’d recommend going this route. It really simplifies the cable management.

Assuming you’ve got all the equipment and the OS installed you’ll want to go through some prep work.

  • Disable swap and remove it from /etc/fstab:
    • From the command line run: sudo swapoff -a
    • Remove the swap mount from /etc/fstab
      From this:

      /dev/sda1 / ext4 defaults 1 1
/dev/hda1 /usr ext4 defaults 1 1
/dev/sda5 swap swap defaults 0 0

      To this:

      /dev/sda1 / ext4 defaults 1 1
/dev/hda1 /usr ext4 defaults 1 1
  • Remove Docker if it’s already present:
    • sudo apt-get remove docker docker-engine docker.io containerd runc
    • sudo snap rm docker
  • Ensure the docker repository is added and enabled. This sounds unintuitive, why would we want to make sure the docker repo is added if we’re removing docker and using containerd? For Ubuntu, containerd is distributed by Docker, not the containerd project. You can read about it here.
    • Install pre-requisites: sudo apt update && sudo apt install ca-certificates curl gnupg lsb-release
    • Ensure the keyrings directory exists: sudo mkdir -p /etc/apt/keyrings
    • Add the Docker repo keyring: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    • Add the Docker repo: echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  • Install containerd
    • sudo apt update && sudo apt install -y containerd.io
    • Ensure the containerd config directory exists: sudo mkdir -p /etc/containerd
    • Ensure the containerd config has the default values: containerd config default | sudo tee /etc/containerd/config.toml
    • Ensure the required modules are loaded at boot: 
      cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
      • If you’re going to try to run things before rebooting, go ahead and modprobe those two modules.
        • sudo modprobe overlay br_netfilter
    • Restart the containerd service: sudo systemctl restart containerd
  • Enable cgroups limit support
    • sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt
  • Configure IPTables to see bridged traffic 
    • cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf 
net.bridge.bridge-nf-call-ip6tables = 1 
net.bridge.bridge-nf-call-iptables = 1 
EOF
    • Apply the configuration without requiring a reboot sudo sysctl --system

Kubernetes time!

Keep in mind that all the example commands here are for Ubuntu 20.04. I’ll leave notes where things need to change for newer versions.

  • Add the Kubernetes repo keyring:
    • curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
  • Add the Kubernetes repo: 
    • cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list 
deb https://apt.kubernetes.io/ kubernetes-xenial main 
EOF
      • If you are using a newer version of Ubuntu, check the official docs to see if there’s a new repo. For Ubuntu 18.04 – 22.04 the kubernetes-xenial repo is what you use, after 22.04, there may be new repos.
  • Install kubelet, kubeadm, and kubectl
    • sudo apt update && sudo apt install -y kubelet kubeadm kubectl
  • Mark the packages as held. We don’t want them to automatically update as there’s a specific upgrade procedure you need to follow when upgrading a cluster.
    • sudo apt-mark hold kubelet kubeadm kubectl
  • Configure containerd as the container runtime:
    • Edit /var/lib/kubelet/kubeadm-flags.env with your editor of choice and add the containerd runtime flags.
    • KUBELET_KUBEADM_ARGS="--pod-infra-container-image=k8s.gcr.io/pause:3.9 --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
      • The important bit is --container-runtime-endpoint=unix:///run/containerd/containerd.sock
  • Ensure forward_ipv4 is enabled:
    • If cat /proc/sys/net/ipv4/ip_forward does not return 1, run the next command
    • # as root
echo 1 > /proc/sys/net/ipv4/ip_forward

Configuring the control-plane node

At this point all of your nodes should have the required software installed. Now we can start building the cluster. There are essentially two types of nodes: master/control-planes and workers. You’ll see control-plane nodes referred to as master as well — I think it’s new terminology to move away from certain terms, but I might be wrong there. master/control-plane nodes handle management of the cluster. What containers run on which nodes. Typically, control-plane nodes don’t run your application containers, just infrastructure-level ones. They’re busy and important, ya know.

  • Generate a token:
    • TOKEN=$(sudo kubeadm token generate)
    • Store this somewhere if you want. I have, but I’ve never run into a reason where I needed it again.
  • Initialize the cluster:
    • sudo kubeadm init --token=${TOKEN} --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint "<your control-plane ip>:<port>" --upload-certs
      • Make sure you update your --control-plane-endpoint to the IP of the control-plane node you’re on.
      • You can configure the --pod-network-cidr to whatever you want, as long as it doesn’t already exist on your network. For most people the default should work.
      • Take note of the join commands it outputs, you’ll use these later.
        • The output should look something similar to this: 
          • Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  /docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
        • For control-planes it should look something like this: 
          • kubeadm join 192.168.2.114:6443 --token zqqoy7.9oi8dpkfmqkop2p5 \ 
    --discovery-token-ca-cert-hash sha256:71270ea137214422221319c1bdb9ba6d4b76abfa2506753703ed654a90c4982b \ 
    --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
          • If you don’t see a command for joining a control plane, you can construct your own with the following: 
            • kubeadm init phase upload-certs --upload-certs
kubeadm token create --print-join-command
            • Take the output of each command and construct your join command by taking the ouput from kubeadm token create --print-join-command and appending control-plane --certificate-key <key-value>
        • For workers it should look something like this: 
          • kubeadm join 192.168.2.114:6443 --token zqqoy7.9oi8dpkfmqkop2p5 \ 
    --discovery-token-ca-cert-hash sha256:71270ea137214422221319c1bdb9ba6d4b76abfa2506753703ed654a90c4982b
  • Make sure you can run kubectl commands without having to be root or use sudo everywhere 
    • mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
      • This will copy the administrative config to your user. This shouldn’t be openly shared.
  • Install flannel as the CNI — Container Network Interface:
    • curl -sSL https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml | kubectl apply -f -

Joining subsequent control-plane nodes to the cluster

Once the main control-plane node has initialized the cluster, you can start joining nodes to it. Depending on the type of node you’re joining, you’ll use different commands. Refer back to the join commands kubeadm init gave you. You’ll run that on each control-plane node as root or with sudo in your cluster.

Joining the worker nodes to the cluster

On each worker node, run the join command kubeadm init gave you as root or with sudo.

Validating your cluster

Now that you’ve joined all your nodes, you can check to make sure they’re all reporting back correctly and are in a Ready state!

kubectl get nodes -o wide

The output should look something like this:

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kube-node1 Ready control-plane,master 7d20h v1.27.3 192.168.10.22 <none> Ubuntu 20.04.6 LTS 5.4.0-1089-raspi containerd://1.6.12
kube-node2 Ready control-plane,master 7d20h v1.27.3 192.168.10.23 <none> Ubuntu 20.04.6 LTS 5.4.0-1089-raspi containerd://1.6.12
kube-node3 Ready <none> 7d19h v1.27.3 192.168.11.194 <none> Ubuntu 20.04.6 LTS 5.4.0-1089-raspi containerd://1.6.12
kube-node4 Ready <none> 7d19h v1.27.3 192.168.11.187 <none> Ubuntu 20.04.6 LTS 5.4.0-1089-raspi containerd://1.6.12
kube-node5 Ready <none> 7d19h v1.27.3 192.168.11.179 <none> Ubuntu 20.04.6 LTS 5.4.0-1089-raspi containerd://1.6.12
kube-node6 Ready <none> 7d19h v1.27.3 192.168.10.244 <none> Ubuntu 20.04.6 LTS 5.4.0-153-generic containerd://1.6.12

Assuming all goes well, you’re ready to start deploying to your cluster!

Using Squid Reverse Proxy to manage multiple domain names on pfSense

In the past, in order to host multiple domain names or sub domains from my homelab, I’ve resorted to running each application on a separate port. This becomes quite cumbersome once you’ve got a small handful of sites that all need external access.

To resolve this, I found multiple sites online suggesting a reverse proxy. So today, we’re going to cover how to implement the Squid Reverse Proxy on pfSense.

 

Step 1 – Adding the Squid package

First things first, we’ll need to add the Squid package if you don’t already have it installed.

Step 2 – Enabling Squid

Next we’ll want to make sure the Squid Proxy itself is enabled, otherwise the Reverse Proxy won’t work.

On the General Tab, Ensure Enable Squid Proxy is checked.

Step 3 – Configuring the Reverse Proxy

Third, we’re going to do a quick set up of the Reverse proxy.

On the General Tab, set the following:

  • Squid Reverse Proxy General Settings
    • Reverse Proxy Interface(s) – Select the interfaces you want the proxy to run on. Typically it’ll just be your WAN interface.
    • External FQDN – The Fully Qualified Domain Name which you’ll be proxying for by default.
  • Squid Reverse HTTP Settings
    • Enabled HTTP Reverse Proxy – checked
    • Reverse HTTP Port – unless you have special needs, leaving this at 80 is fine.
    • Reverse HTTP Default Site – This is the default sub domain you want to redirect to if there’s nothing found in the mappings (we’ll cover that later)
  • Squid Reverse HTTPS Settings
    • Enable HTTPS Reverse Proxy – checked
    • Reverse HTTPS Port – unless you have special needs, leaving this at 443 is fine.
    • Reverse HTTPS Default Site – This is the default sub domain you want to redirect to if there’s nothing found in the mappings (we’ll cover that later)
    • Reverse SSL Certificate – This is the cert to use for the domains you want to use. In this example it’s using the wild card cert which we created in this post.
  • Click Save

Step 4 – Adding Web Servers

Go to the Web Servers tab and click ‘Add’.

Enter the internal information for the web server you want to direct traffic to.

Do this for each web server and protocol you need. For example, if you have a web server that hosts on both ports 80 and 443, you’ll want to add two web servers.

Step 5 – Adding Web Server Mappings

On the Mappings tab, click ‘Add’.

Fill out the mapping information, for the site in question. If you added multiple web servers for the same physical IP/machine, you can select multiple “peers” — aka Webservers — here.

Under the URI setting, add as many patterns as you need for the proxy to use as criteria to map to the set of servers.

For example, if you have HTTP and HTTPS sites running on the same server, you could add both:

  • ^https://external.example.com/.*$
  • ^http://external.example.com/.*$

After you’ve clicked ‘Save’ you should be off to the race!

Creating Wildcard Certificates on pfSense with Let’s Encrypt

As we progress in the internet age, we put more and more emphasis on security. This has previously been a bit more difficult to do for someone who hosts their own sites and services. Dealing with self-signed certificates or having to shell out hard earned cash for a “legit” cert has always been a hassle. More recently however — the last few years –, this has become a lot more attainable — especially to the homelabber — to create fully trusted certificates without all the headache of having to purchase them from a “trusted” party. Enter Let’s Encrypt, a service which allows anyone to obtain certificates for free.

“Great, Let’s Encrypt, yes yes, we’ve all heard about it. The title says wildcard certs on pfSense, get to the good stuff!”, yea yea, I hear ya.

In this article I’m going to cover how to add an ACMEv2 Account Key, and a wild card cert using the ACME package in pfSense.

Prerequisites:

  • A pfSense installation
    • In this article I’ll be showing you how to do this on pfSense version 2.4.4-RELEASE-p3 .

Step 1 – Adding the package

First thing you’ll want to do is make sure you have the ACME package installed.

From the Package Manager screen go to Available Packages and search for and install “acme”. Once it’s installed it will show up on your Installed Packages list.

Step 2 – Register your Account Key

Once you’ve gotten the package installed, you’ll want to register an account key with Let’s Encrypt.

Under Services, go to Acme Certificates.

Go to the Account keys tab, and click “Add”.

Fill out the form ensuring you select “Let’s Encrypt Production ACME v2” from the ACME Server drop down. If you’re in a testing environment, or want to test certificates out you can select the “Staging” server instead. This helps when you’re having issues with your certs and don’t want to run into the rate limits that are imposed on the production servers.

Also make sure you enter a valid email address you have access to, this will be the address all notifications go to in regards to the certificates you create using this key.

After you’ve fill everything out, click “Create new account key” and then click “Register ACME account key”. Once that’s been successfully completed, you’ll get an Account Key in the Account Key field.

Lastly, click “Save”.

Step 3 – Add your Wildcard Certificate

Now that we have an account key, we can start creating certificates.

Go to the Cerficates tab and click “Add”

Fill out the form making sure you choose the ACME v2 Account Key you created in the previous step.

Under the Domain SAN list, you’ll want to add two entries for each domain you want a wildcard for. One plain, one with the wildcard.

For example, if I wanted a wildcard domain for *.myexample.com, I’d have to add both myexample.com and *.myexample.com.

This is also where things differ between setups. Depending on where your domain names come from, and what process you want to use, you’ll follow different directions. I, for example, have my domains through GoDaddy. Using the DNS Method is quite easy and doesn’t require you to have certain ports open (ie: port 80).

Under action list, you can specify things to happen after a cert is issued or renewed. I have mine set to run scripts which copy the certs out to my various web servers and recycle the server daemons. You can do whatever you like 😉

Fill out the required fields for your provider — remember that you’ll have to add two domain entries at a minimum — and then click “Save”.

Step 4 – Set the General Settings (Optional)

If you want to do something with your certs after they’ve been issued/renewed (Actions under the Action List in the previous step), writing them to a location on the pfSense server is convenient.

To do this, on the General tab, check the “Write Certificates” option and click “Save”.

Step 5 – Issue the Certificate

Now we’re set to get some certs!

Under the Certificates tab, you should see your newly created cert, click the “Issue/Renew” button. This will start the process of requesting a cert and validating that you have ownership of the domain. Once the process completes you’ll be prompted with a big green wall of text. Be sure to read it carefully! Even though it’s green and the top may say success, there could be errors listed that you’ll want to resolve.

 

Now that you’ve got your cert, you’ll probably want to host a bunch of stuff! I’ll cover that in my next post, Reverse Proxy with Squid.

Internet Status Monitoring with Home Assistant and Node-RED

If you’ve read previous posts on my blog, you’ll undoubtedly know that I’ve had some issues with my internet connection with Comcast. In the last few years the service has been fairly decent, especially in the cooler months. But for some reason when it gets hot outside (90F+ on average) the connection drops throughout the day. My suspicion, and I have no actual proof to back this up outside of the temperature, is that there’s a piece of hardware in an equipment shed somewhere that’s overheating, or becoming unstable due to inadequate cooling. I’ve tried customer service, but, as expected got nowhere with them. They either apologized and said there’s really nothing they can do, or they’d try to “push a fix to my modem”. Even after explaining that there have been multiple techs out to the house only to find that there’s nothing wrong either in the house or to the pole on the street. The only thing I seem to be able to do is complain and get nothing resolved.

Initially, this project was just to help automate the rebooting of my modem. But after I was seeing 20+ disconnects a day, I decided to also incorporate tweets into the solution.

Prerequisites

    • Home Assistant with the following
    • A controllable switch which is connected to your modem
      • I use a GE Z Wave Switch, but anything that’s integrated with Home Assistant should work.   

Home Assistant – Switches and Sensors

In Home Assistant, you’ll need to do the following:

    1. Ensure that you can control your switch. Depending on the kind of switch you’re using the configuration and setup will be different, so I’m doing to skip that in the post.
      1. The home assistant docs, and the home assistant and home automation reddit pages are quite helpful.
    2. Create a binary sensor which evaluates if the internet is up or down.
      1. In your configuration.yaml file, add a binary_sensor: entry if you don’t have one, otherwise append to the existing one. 
        1. binary_sensor:
  - platform: ping  
    host: google.com  
    count: 5  
    scan_interval: 30  
    name: "Internet Status"
           
        2. You can choose whatever domain you like you ping. I just chose google as they’re probably up more than Comcast is /s

Node-RED – What do do when the connection drops

The flow isn’t all that complicated. I’ll give a quick overview before we dive into specifics.

    • When the Internet Status goes from 1 to 0
    • Start a 2 minute timer 
      • Sometimes there are hiccups.
    • Check the internet status again
    • If the status is still 0 (down), then turn off the power to the modem
    • Start a 30 second timer
    • Turn the power to the modem back on
    • Check the Internet Status every 10 seconds until it’s back up
    • Compose the message for the tweet
    • Replace the datetime token with the current datetime
    • Send the Tweet

Step 1 – Triggering on Status change

Use a State Trigger to detect that the Internet has gone down.

Only trigger if the state has gone from “on” to “off”.

Step 2 – Wait to make sure it’s actually down

Sometimes the connection will hiccup and show as down. To combat this, we’ll just wait a bit before we go through the whole process of “Power Down, Power Up”, which could take up to 5 minutes to complete (for me anyway). So we’ll add a stoptimer and set it to 2 minutes. This can be changed to your preferences, but I think it’s a good amount for my needs.

Step 3 – Check the Internet Status

We’ll use the Current State function node to check the Internet Status. Once you’ve set the properties, the node should have two outputs. One if “state is on” is false, and the other if “state is on” is true. We’ll link Step 3 to Step 4 using the “If State is False” output.

        

Step 4 – Turn off the Modem

To turn off the modem we’re going to use the Service Call function.

And we’re going to tell Home Assistant to turn off the switch associated with the modem. I happen to have renamed mine to switch.ge_media_room_modem_switch.

Step 5 – Wait a bit for you know, reasons.

We’ll use another stop timer to wait before we turn the modem back on. 30 seconds sounds good, right? It’ll be the same as step 2, add another Stop Timer except this time we’re only running for 30 seconds.

Step 6 – Delete the payload message

In order to turn things off and on in the same flow, we’ll need to delete the current payload message, otherwise Node-RED will just use the same payload value as before (turn_off). To achieve this, we’ll use the Change function node.

Step 7 – Turn the modem back on

Now that we have a fresh empty payload again, we’ll use another Call Service node and tell Home Assistant to turn the power back on.

Step 8 – Wait a bit before checking the Internet Status

In order to send out tweet, we’ll need to have an internet connection that’s working. So now we wait. For about 10 seconds. Add another Stop Timer and set it for 10 seconds

Step 9 – Check the Internet Status

We’ll use the Current State function node again to check the Internet Status. This time, we’ll link the “false” output back to the input of the timer in step 8, the true will continue on to the input of the node in step 10.

        

Step 10 – Set the Tweet message

We’ll use another Change function node here, but instead of just deleting the payload message, we’ll be setting it to a specific value; our tweet. You can tweet whoever you want, I tweet the Comcast support account.

Here’s my message:

Hi @ComcastCares, I just wanted to let you know that the internet went down again and i had to reboot my modem at %timestamp%. #downdetection #automation #terribleservice #homeassistant

Step 11 – Replace the timestamp token in the message (Optional)

Since my message contains a timestamp token (“%timestamp%”) I want to send along with the tweet, I need to be able to get the current datetime. I do this with a function node and run a tiny bit of javascript.

var date = new Date();
msg.payload = msg.payload.replace("%timestamp%", date);
return msg;

Step 12 – Send the tweet!

To send the tweet we’ll use the Twitter OUTPUT function node. Now we can send the tweet! Well, once you’ve configured your credentials. 

If your properties window looks like this, then you’ll need to set your credentials first. Click the pencil to get to the next screen.

From here you’ll set your twitter account you want to use, and then after following the steps in step 1, enter in your api and access token information.

 

Once you’ve done that, your user should show in the list and you can start sending tweets!

 

Blog Migration

Hi. It’s been a while. This is pretty much par for the course though. I just wanted to make an update since I’ve spent pretty much all day moving this blog from my old hosting company to a local WordPress install in my homelab. No point in paying for hosting something that I barely update or do anything with. As such, don’t be surprised if there are some broken links, or missing pages. I’m attempting to grab everything from the old host but, due to one thing or the other, it’s difficult to discern what’s actually needed and what’s just junk.

While working on this migration I was able to take a trip down memory lane and reminisce about posts of the past, as well as get inspired to make a post or two about some topics that were covered in the past (looking at you auto-tweet-when-the-internet-goes-down script).

So, check back from time to time and maybe you’ll see something new!

Modem Monitor Update [Updated]

I bought a new modem today, slightly hoping that it will fix whatever connection issues i’m having, it’s a newer model surfboard (SB6190) and as such I’ve had to update my script which checks the status.

import requests
import datetime
from bs4 import BeautifulSoup
#import dateparser
import tweepy
import re

#Set up Twitter creds
CONSUMER_KEY='[REPLACE ME]'
CONSUMER_SECRET='[REPLACE ME]'
ACCESS_KEY='[REPLACE ME]'
ACCESS_SECRET='[REPLACE ME]'

#Authenticate with Twitter
auth=tweepy.OAuthHandler(CONSUMER_KEY,CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY,ACCESS_SECRET)
api=tweepy.API(auth)

#Set up regex patterns
timespanPattern = re.compile("^(\d+)\s+d:\s+(\d{1,2})\s+h:\s+(\d{1,2})\s+m$")

#scrape the modem page
page = requests.get("http://[REPLACE WITH INTERNAL IP OF MODEM]/cgi-bin/swinfo")
soup = BeautifulSoup(page.content, 'html.parser')

#navigate the DOM to get the desired data
modeminfo = list(soup.find_all('table', {"class": "simpleTable"})[1])
#print modeminfo
uptime = list(modeminfo[3].children)[3]
#print uptime
timestr = uptime.get_text()
#print timestr

#Take the text and get the time data out
matches = timespanPattern.match(timestr)
#print "Days: " + matches.group(1)
#print "Hours: " + matches.group(2)
#print "Minutes: " + matches.group(3)
days = int(matches.group(1))
hours = int(matches.group(2))
minutes = int(matches.group(3))

#create timespan so we can figure out at what time the reboot happened
span = datetime.timedelta(days=days,hours=hours,minutes=minutes)
currenttime = datetime.datetime.now()
timespan = currenttime - span
status = ""

if minutes < 5 and days == 0:
status="Hey @Comcast, my modem just rebooted at " + timespan.strftime("%Y-%m-%d %H:%M:%S")
api.update_status(status)
print status
else:
print "Modem has been up since " + timespan.strftime("%Y-%m-%d %H:%M:%S")

Update 7/15/2017 19:04
It looks like the new modem doesn’t behave like the old one. The old one would reboot automatically when things got unstable and would log a bunch of T3 and T4 errors. The new one does similar, but doesn’t reboot, it’ll just stay up. I’ll have to find a new approach to annoy Comcast 😛 Today the internet connection wouldn’t stay up for longer than an hour after I put the new modem on. I got in contact with Comcast’s online support and they did some magic on their end and after a few modem reboots and reboot of my firewall, things seem to be more stable. At this point the connection has been up for ~2 hours without incident. Here’s to hoping we’ll get a few weeks more of uptime 🙂

Comcast can’t keep it up? I’ll let them know about it

So over the last few months my internet service with Comcast has been less than stellar. My connection would drop out at random times throughout the day, on some days I wouldn’t stay connected for more than 20 or so minutes at a time. At first, I was hoping that it was a random issue and it would go away on its own (wishful thinking, right?). Well, it didn’t. I contacted customer support and they told me that my modem levels were high, but they couldn’t do anything about it remotely and that a tech would have to come out. I find this weird because last year I was having a similar issue and the rep was able to resolve my problems remotely. Anyway, I scheduled an appointment and a tech came out and took a look at my equipment and was unable to find anything wrong. The only thing he said may be causing an issue was that I’d put in a splitter that wasn’t “one of theirs”. After he left i checked mine against his, and the frequency ranges on both the splitters was identical.. so.. whatever. Needless to say, my issue was not resolved. So a few days later when the issue cropped up again, I scheduled another appointment for a tech to come out. This time the guy busted his butt (much respect!) and ran brand new cable from the pole to the house, cleaned up the rats nest of cabling the initial installers left behind when we first got service and even put in an outdoor service box.

This was a few weeks ago, and up until the last few days the problem seemed to have been resolved. But then it started again. And I was about to schedule another appointment, but I’d gotten my bill that morning. Come to find out they charge you to come out and fix the issues they can’t seem to keep resolved. $30 bucks a pop! Needless to say, before I give them another excuse to gouge me for more money, I figure some internet noise is required.

I’ve decided to write a small python script that scrapes the uptime from my modem (an Arris Surfboard SB6141), and if it’s less than 5 minutes, pop a tweet at Comcast letting them know. Feel free to use this or improve upon it!

The script runs from an Ubuntu server every 5 minutes via crobjob.

Dependencies (installed with pip)

  • requests
  • datetime
  • BeautifulSoup
  • dateparser
  • tweepy

Script

import requests
import datetime
from bs4 import BeautifulSoup
import dateparser
import tweepy

CONSUMER_KEY='[REPLACE ME]'
CONSUMER_SECRET='[REPLACE ME]'
ACCESS_KEY='[REPLACE ME]'
ACCESS_SECRET='[REPLACE ME]'

auth=tweepy.OAuthHandler(CONSUMER_KEY,CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY,ACCESS_SECRET)
api=tweepy.API(auth)

page = requests.get("http://[REPLACE WITH INTERNAL IP OF MODEM]/indexData.htm")
soup = BeautifulSoup(page.content, 'html.parser')

modeminfo = list(soup.find_all('table')[1].children)
uptime = list(modeminfo[1].children)[5]
timestr = list(uptime.children)[3].get_text()
timespan = dateparser.parse(timestr)
currenttime = datetime.datetime.now()
span = currenttime - timespan
minutes = span.seconds/60
days = span.days
status = ""

if minutes <= 5 and days == 0:
  status="Hey @Comcast, my modem just rebooted at " + timespan.strftime("%Y-%m-%d %H:%M:%S")
  api.update_status(status)
else:
  print "Modem has been up since " + timespan.strftime("%Y-%m-%d %H:%M:%S")

 

Setting up Comchap’s Comcut with ComSkip, ffmpeg, and HandbrakeCLI

I’ve been running Plex for years now and with their addition of Plex DVR last year I started recording many of the television shows I’d been acquiring other ways before. It saves on bandwidth, it saves on time spent searching for the right quality from a trusted group, but what it doesn’t save on is space. An hour show can easily run 6GB if recorded at 1080p. To help with that I’ve implemented a two stage strategy.

  1. Remove the commercials, which take up almost a third of a show.
  2. Compress the show a bit with x264 using Handbrake.

First we’ll need to cover exactly what we’re going to need:

  1. A linux server which will do the compression for you — I’m running Ubuntu
  2. git — we’ll be getting a few libraries from github repos.
    1. ComSkip: https://github.com/erikkaashoek/Comskip
    2. ComChap: https://github.com/BrettSheleski/comchap
  3. enough hard drive space to do all this — 20GB should work for a work space.

Ok, so first things first. We’ll want to make sure that your server is set up to build code as well as some other dependencies.

For my version of Ubuntu I installed the following:

$ sudo apt install -y git handbrake-cli build-essential libargtable2-dev libavformat-ffmpeg-dev libsdl1.2-dev autoconf automake libass-dev libfreetype6-dev libsdl2-dev libtheora-dev libtool libva-dev libvdpau-dev libvorbis-dev libxcb1-dev libxcb-shm0-dev libxcb-xfixes0-dev pkg-config texinfo zlib1g-dev

Next, we’ll want to ensure we have a version of ffmpeg that will work. I’ve personally run into issues using ffmpeg which shipped with my release of Ubuntu (I’m running 16.04.02). The simplest way that I’ve found to get around this is to compile ffmpeg from source. I followed the directions here: https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu

Some things to keep in mind. If you’re going to need to download the sources for any of the dependencies in that guide instead of just installing them via the package manager, you’ll probably want to make sure that you’re getting the latest version of that library instead of what’s listed in the guide. Some of those libraries seem quite old.

After ffmpeg is installed, it should reside in your users’ ~/bin folder. If you want, you can link/copy it to somewhere more accessible if more than that user need to use it (something like /usr/local/bin?)

Now we’re ready for the main event.

First, grab ComSkip from the github repository:

$ cd ~/
$ git clone git://github.com/erikkaashoek/Comskip
$ cd Comskip
$ ./autogen.sh
$ ./configure
$ make

This will create a comskip executable in the Comskip directory. For other applications to use it we’ll need to move it into a directory which is typically in everyone’s $PATH. I copied mine to /usr/local/bin/comskip. Remember to check and make sure it’s executable (chmod +x comskip)

Second, we’ll grab the Comchap library from github as well:

$ cd ~/
$ git clone git://github.com/BrettSheleski/comchap

Third, we’ll need to get a script to tie this all together.

$ nano convertTStoMKV.sh

#!/bin/bash
PATH=$PATH:/usr/local/bin
searchDir=$1
fileExt="ts"

if [ -d "$searchDir" ]
then
find "$searchDir" -name "*.$fileExt" -type f -exec sh -c '
file="$0"
fileDir=$(dirname "$file")
fileName=$(basename "$file")
fileNameNoExt="${fileName%.*}"
tmpFileDir="/your/path/to/the/working/directory"
newFileExt="mkv"
newFileName="$fileNameNoExt.$newFileExt"
workingFilePath="$tmpFileDir/$fileName"
tmpFilePath="$tmpFileDir/$newFileName"
finalDestination="/your/path/to/where/the/file/is/dropped"
echo "moving file to work dir: $file"
mv "$file" "$workingFilePath"
if [ $? -eq 0 ]
then
echo "Running ComCut on file: $workingFilePath"
/your/path/to/comchap/comcut "$workingFilePath"
echo "processing: $workingFilePath"
HandBrakeCLI -e x264 --encoder-preset "faster" -q 24 -2 -T -E copy -B 192 -5 -i "$workingFilePath" -o "$tmpFilePath"
if [ $? -eq 0 ]
then
echo "Moving file to final destination: $finalDestination/$newFileName"
mv "$tmpFilePath" "$finalDestination/$newFileName"
if [ $? -eq 0 ]
then
echo "Removing old file: $workingFilePath"
rm "$workingFilePath"
fi
fi
fi
' {} ';'
else
echo "dir doesn't exist: $searchDir"
fi

Remember to update all the relevant paths. I’ve made some of them generic to explain what they are. Once that’s done, then set up the script to run at some point every night (I used cron) and all the new .TS files will have their commercials cut, be compressed and converted to .MKV files.

Pre Paid Progress

It’s been about two months since i dropped Sprint and went to T-Mobile. My experience has been pretty good so far, but there are a few ‘but’s in there. For the transfer I got an unlocked Nexus 4. The plan is that here in a while i’ll also get an AT&T prepaid plan and compare the services.

Price

I was able to get in on the $30/month prepaid plan for T-Mobile. Which is even cheaper that what I was going to get through Straight Talk. This plan is just rediculous! You get unlimited data and texting (up to 5GB at 3/4G speeds) and 100 minutes of talk. I really don’t talk on the phone much so that’s just fine for me, and I don’t think I’ve come close to hitting the 5GB limit on data (…yet ;))

Speeds

Where i live and with my phone, the fastest I’ll get is HSPA+42 (T-Mobile’s non-LTE 4G). In town, and once i’m in other towns I’ve gotten good speeds of between  1200 and 6000 kbps down, which is a huge increase from the 30 – 500 kbps i was getting with Sprint.

Coverage

This is where i think T-Mobile fails a bit. While the only place i’ve never had service was the same place most other people didn’t have service, there are areas where i wished me speeds were better. While in town i have great service and good speeds, but if i’m on the interstate or travelling between towns, often times my connection drops to E or even G — at which point doing any kind of music streaming is just impossible. Overall I would give the coverage a grade of satisfactory, but with much room for improvement.

With that said, I’m definitely more content with T-Mobile than I was with Sprint. I am curious though on how AT&T will stack up. I have friends on AT&T and while they have more and better coverage, I don’t think they get the same speeds that i do. I’d like to do an Apples-to-Apples comparison in the next few months.

Trying to move from Sprint to Straight Talk

This whole process has been quite frustrating.

But first, let me give you some background information on what i’m trying to do. I’ve been with Sprint for years, I’d say since 2000 or 2001 when i was still on with my parents. Then a few years ago, 2008 i think, I got a new phone and wanted to go off on my own. So i got a new number and my own account with Sprint. For the longest time I was quite happy. The speeds were good, the call quality was great, and the phones were pretty on par with everyone else. But then the smartphone revolution happened. And slowly i started to notice that Sprint had fewer and fewer phones that i was actually interested in. I also started noticing that my speeds weren’t keeping up with what other carriers were offering. So finally, after my contract expired in august i’ve been on the lookout for what new phone i’d want to get, and what carrier i’d go to.

Then a few months ago I discovered the Nexus 4. I knew about it for a while, but i didn’t quite looked into it. What really caught my eye is that it’s 1) a newer phone and 2) you can get it unlocked cheaper than you can get it from a carrier! The unlocked portion is extra nice especially since it’s now illegal for you to unlock your own phone!

Now that i knew what phone i wanted to get it was time to start looking at providers. A friend of mine has been using multiple resellers (or MNVOs) over the last two years or so and has had some good things to say. He was on Straight Talk at the time using AT&T. I figured I could get a couple of SIMs and then test out both networks, since Straight Talk used both AT&T and T-Mobile networks. I’d also looked at a few others and while the prices were comparable, I noticed that only Straight Talk explicitly referred to offering 4G speeds (which for the Nexus 4 meant HSPA+).

I order my phone from Google, and my SIM and Service Plan from Straight Talk. When the phone comes in i’m totally stoked! It’s pretty and clean and fast and everything i’d hoped it would be! I wait a few days and get my SIM (on 4/11). First thing i noticed was that my SIM was a full sized one, and not a micro SIM which the Nexus 4 needs. After hunting around and even stooping to go to Walmart to try and get a different SIM (which they don’t have for ST), i found a ‘conversion’ kit i could print out online. I chop up my SIM to make it fit into the SIM Slot, turn on the phone and everything looks good. It detected the SIM and started looking for signal. I then went to the Straight Talk website to Activate my SIM and service. Everything goes pretty smoothly from what i can tell. I say that i’m activating a new SIM and that i want to transfer an existing number from another company. I put in all the information they requested for my Sprint account, and then completed the process. I noted that for company to company transfers it could take up to two business days (this is weird because I later read online that there’s an FCC regulation saying that these transfers are supposed to be done within 24 hours).  I wait until the next Monday (4/15) and then make a post on the Straight Talk facebook page, which I’d read was the best way to get any kind of customer service. I ask if i can get some assistance since my port had taken longer than the two days they communicated in their policy. The conversation starts with comments on my post asking me to put in a helpdesk ticket, which i try, but the page kept giving me an error saying that my email address belonged to a deleted account. In the end we resort to Facebook Messaging, which I’ve exported and will display here fully intact except for any sensitive information. I find the whole situation quite frustrating.

On April 15, 2013 8:43:35 AM PDT, Dan Colomb wrote:
Hi Phil,
 The MEID of my old Sprint Phone is XXXXXXX, the IMEI of my new phone is XXXXXXX.
my ST user account email is xxxx@xxxx.com
Let me know if you need anything else.
On April 15, 2013 8:44:23 AM PDT, Phil ST wrote:
Thank you for that information, Dan.
Let me check your account now. I'll get back to you in few minutes.
On April 15, 2013 8:45:09 AM PDT, Phil ST wrote:
Can you provide me the phone number that you're trying to transfer to Straight Talk?
On April 15, 2013 8:46:44 AM PDT, Dan Colomb wrote:
555-555-5555
On April 15, 2013 8:53:13 AM PDT, Phil ST wrote:
Were you given a reference or ticket number for your port request? If so, please provide me with the numbers so that I can pull out your account. Thanks!
On April 15, 2013 8:57:16 AM PDT, Dan Colomb wrote:
The only thing i see on my ST account is the SIM (XXXXX) and that it states there is a port in progress. I don't recall receiving an email with a reference or ticket number.
The only number i have in my email is for my SIM order (XXXXX).
On April 15, 2013 9:10:55 AM PDT, Phil ST wrote:
Dan,
We regret to inform you that we failed to port your number to the SIM card that you purchased. If you want we can activate your SIM with a new phone number or you may purchase a brandnew Straight Talk phone and we'll attempt to port again the phone number. Thanks!
On April 15, 2013 9:17:26 AM PDT, Dan Colomb wrote:
Wow, that is not the response i was expecting. Is there a reason why the port failed?
I noticed that on your website you no longer offers Micro SIMs, that is what my phone requires. How would i go about getting a Micro SIM for ST?
On April 15, 2013 9:20:05 AM PDT, Phil ST wrote:
Because porting any numbers to a T-Mobile compatible SIM is not possible as of now.
For now, Straight Talk Wireless is not selling SIM cards that are compatible with AT&T devices. We will bring them back as soon as we can. For updates on this issue, please visit our website at www.Straighttalk.com.
On April 15, 2013 9:22:41 AM PDT, Dan Colomb wrote:
Is there a way to get a refund for the 30 days of service i purchased? I guess i'll be going to another provider.
On April 15, 2013 9:23:45 AM PDT, Phil ST wrote:
We're sorry to hear that, Dan.
Where did you purchase your service plan?
On April 15, 2013 9:24:08 AM PDT, Dan Colomb wrote:
I purchased it through the website when i bought the SIM.
On April 15, 2013 9:25:10 AM PDT, Phil ST wrote:
Dan,
If you want, you may purchase a brandnew Straight Talk phone and we'll just convert the amount you paid for the SIM and will add also your service plan.
On April 15, 2013 9:27:55 AM PDT, Phil ST wrote:
In addition, we will also re-process the porting of number if ever you will purchase a Straight Talk phone.
On April 15, 2013 9:28:44 AM PDT, Dan Colomb wrote:
i don't want to buy a new phone, i want so use the new Unlocked GSM phone which i just purchased.
So if, as of right now, there is no way for me to get my number transferred and have my service activated, i would like to have my 45 dollars refunded. I can handle eating the SIM price, since i know that's non-refundable.
On April 15, 2013 9:37:09 AM PDT, Phil ST wrote:
Dan,
I'll need first to update my floor supervisor regarding your request. I'll get back to you in few minutes. Thanks.
On April 15, 2013 10:48:33 AM PDT, Phil ST wrote:
Dan, 
We regret to inform that we do not issue refunds for airtime purchases.

This did not go anywhere near where i thought this would go. So i’m out about 70 dollars, and have nothing to show for it. Like a told a friend, i went from irritated, to insulted, to angry. I’m totally disappointed in the customer support, and will never use Straight Talk again. And i urge you all to stay away from them.

On the plus side, i ended up going to the T-Mobile store during lunch and found out that there’s a chance i can still get on their old $30 dollar a month unlimited pre-paid plan, which i may be able to go to Walmart (yes, another trip to WM… *shudder*) for and get today. Otherwise i can just order a new SIM online, which will properly fit my phone, and get a plan that’s even cheaper than Straight Talk’s. If i can’t get in on the $30 a month plan, then i’ll go to the new play they offer which is $50 a month, only 5 dollars more than ST.

I’ll let you all know how things go 🙂