Old Tech Log

November 16, 2017

Getting started with a simple Java application using **gradle**

mkdir DgraphJavaSample
cd DgraphJavaSample
gradle init --type java-application

Modify the build.gradle file to change the repositories and dependencies:

// Apply the java plugin to add support for Java
apply plugin: 'java'

// Apply the maven plugin to add support for Maven
apply plugin: 'maven'

// Apply the application plugin to add support for building an application
apply plugin: 'application'

// Use maven to pull down dependencies
repositories {
  mavenCentral()
}

dependencies {
  // Use Dgraph Java client
  compile 'io.dgraph:dgraph4j:0.9.2'

  // Use JUnit test framework
  testCompile 'junit:junit:4.12'
}

// Define the main class for the application
mainClassName = 'App'

November 17, 2017

Dark Theme Fix for Firefox

November 26, 2017

Readline Keyboard Shortcuts

Tmux Tips

Kubernetes Networking

Technology Buzzwords for the Next 10 years

Solving display hang issue on T470 This is what shows up in journalctl before the display goes blank. The computer is still operational and it may be possible to get to it using SSH:

Jan 01 11:52:02 cottongeeks kernel: [drm:i915_gem_idle_work_handler [i915]] *ERROR* Timeout waiting for engines to idle
Jan 01 11:52:03 cottongeeks kernel: [drm:i915_gem_idle_work_handler [i915]] *ERROR* Timeout waiting for engines to idle
Jan 01 11:52:48 cottongeeks kernel: [drm] GPU HANG: ecode 9:0:0xfffffffe, reason: Hang on rcs0, action: reset
Jan 01 11:52:48 cottongeeks kernel: [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
Jan 01 11:52:48 cottongeeks kernel: [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
Jan 01 11:52:48 cottongeeks kernel: [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
Jan 01 11:52:48 cottongeeks kernel: [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
Jan 01 11:52:48 cottongeeks kernel: [drm] GPU crash dump saved to /sys/class/drm/card1/error
Jan 01 11:52:48 cottongeeks kernel: i915 0000:00:02.0: Resetting rcs0 after gpu hang
Jan 01 11:53:00 cottongeeks kernel: i915 0000:00:02.0: Resetting rcs0 after gpu hang
Jan 01 11:53:05 cottongeeks systemd-logind[385]: Lid closed.
Jan 01 11:53:05 cottongeeks systemd-logind[385]: Suspending...

I tried setting kernel parameters using grub: https://wiki.archlinux.org/index.php/Kernel_parameters#GRUB and set enable_psr=0 . It supposedly has some implications for battery power, but otherwise seems to ensure the GPU does not crash.

January 7, 2018

On Arch Linux, the tmux socket in the tmp folder keeps getting deleted. I am yet to get to the bottom of that (probably some systemd service that is doing it), but here is a way to get back the socket:

killall -SIGUSR1 tmux

January 8, 2018

Redirecting a non-www website to a www website Let’s take an example:

This is a bit trickier than it initially appears. But it can be achieved with a combination to two S3 buckets and two corresponding CloudFront distributions. Here is a step by step guide:

Installing NPM w/o sudo

WebAssembly

VLC hangs when in full screen

January 20, 2018

Tmux Mouse Mode Tweaks

January 24, 2018

Setting content type on files in S3 w/ no extension

find public -type f ! -name "*.*" -printf '%P\n' | \
 xargs -n 1 -I% aws s3 cp --content-type 'text/html' \
 "s3://www.deepakjois.name/%" "s3://www.deepakjois.name/%" \
 --metadata-directive REPLACE

March 1, 2018

Learning Kubernetes Things to dig into after reading Learning Kubernetes The Hard Way:

EdgeRouter config

Two Simultaneous PPPOE connections

configure
set interfaces ethernet eth0 pppoe0 default-route none
set interfaces ethernet eth1 pppoe1 default-route none
set load-balance group G interface pppoe0 route-test type ping target 8.8.8.8
set load-balance group G interface pppoe1 route-test type ping target 8.8.4.4
set protocols static interface-route 0.0.0.0/0 next-hop-interface pppoe0
set protocols static interface-route 0.0.0.0/0 next-hop-interface pppoe1
commit
save
exit

Disconnect/Reconnect

disconnect interface pppoeX
connect interface pppoeX

Adjust weights

set load-balance group G interface pppoe0 weight 70
set load-balance group G interface eth1 weight 30

March 30, 2018

Load Balancing Guide https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236

April 9, 2018

Installing minikube

Upgrading **pip** packages in bulk

Transcoding AC3 audio in TV shows to MP3 for Mi TV Mi TV does not appear to support AC3 audio. So only the video plays. In order to fix the problem, we need to transcode the audio to MP3 or AAC and keep the rest of the data (video and subtitles) the same:

ffmpeg -i video.mp4 -c:v copy -c:s copy -c:a mp3 video_out.mp4

Wrote a neat little script to transcode files in a folder

!#/bin/sh
set -ex

dir=$(dirname "$1")
base=$(basename -s .mp4 "$1")

ffmpeg -i "$1" -c:v copy -c:s copy -c:a mp3 "$dir/$base.transcoded.mp4"

EdgeRouter bandwidth tests Link to forum post: https://community.ubnt.com/t5/EdgeRouter/ER-X-SFP-router-bandwidth-test-maxing-out-at-200Mbps/m-p/2315375#M204712

Server:

iperf3 -s

Client:

iperf3 -c 192.168.1.2 -w 64K -P 20

April 14, 2018

What happens when … Kubernetes edition!

Using **pipenv** w/ edx-dl After checking out the edx-dl source

pipenv install -e .
pipenv shell

May 2, 2018

Things one can do with IP Addresses

Mounting Samba shares w/ systemd On Arch, I observed that if a Samba share is listed in /etc/fstab, it tries to mount before the network is available.

Here is one possible workaround using x-systemd.automount:

//192.168.1.3/courses /home/deepak/Volumes/courses cifs x-systemd.automount,x-systemd.idle-timeout=1min,username=deepak,password=v00d00d0ll,vers=1.0,uid=deepak,gid=wheel

May 9, 2018

Kubernetes and BGP:

https://www.youtube.com/watch?v=tHAkey-sZ9g&&t=1548s

https://youtu.be/tHAkey-sZ9g?t=1548s Presentation slides: https://schd.ws/hosted_files/kccnceu18/15/Global%20Container%20Networks%20-%20KubeCON%20EU%202018.pdf

Casting any video to Chromecast

Java SDK and Freetype Ran into an interesting problem where the Java SDK I was using bundled an older version of freetype which got picked up by LibreOffice. Here is a way to debug which library is getting picked up by a program

LD_DEBUG=bindings,libs libreoffice --writer

List of Technical Book Recommendations

https://twitter.com/copyconstruct/status/994079362273153024

https://twitter.com/copyconstruct/status/994079362273153024

May 13, 2018

Kubernetes Internals

Using Caddy to serve a web app over self-signed SSL You can also front it with Caddy in a single command, no need to generate the self-signed certificate manually and find a place to store it:

$ caddy -host localhost "proxy / localhost:9000" "tls self_signed"

This will serve your Node application (assuming it’s on port 9000) at https://localhost:2015/. And you can change the port from the default port with the -port flag.

Source: https://news.ycombinator.com/item?id=17062974

May 16, 2018

Update git submodules to latest

git submodule update --remote --merge

From: https://stackoverflow.com/a/42175412/31979

June 5, 2018

Using a different TeX format (for e.g **ufy**) with the same luatex executable

Reference: https://tex.stackexchange.com/questions/124421/how-can-latex-and-pdflatex-be-both-symbolic-links-to-same-executable-pdftex-an

June 7, 2018

Adding subtitles in .srt file to a video file Option 1:

ffmpeg -i movie.mkv subtitle.srt -map 0 -map 1 -c copy output.mkv

Option 2 (using mkvmerge):

mkvmerge -o output.mkv input.mkv subtitle.srt

August 3, 2018

S3 Static Website Hosting

System Design Primer

VirtualBox

SSH Key Auth for first time

ssh-copy-id -i .ssh/id_rsa.pub username@hostname

November 1, 2018

Folder Sharing in VirtualBox Links:

Enable symlinks on Virtualbox shared folder and restart VM (otherwise npm install will not be able to create symllinks). Link: https://askubuntu.com/questions/446317/how-to-make-guest-os-follow-symlinks-from-shared-folder

VBoxManage setextradata VM_NAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/SHARE_NAME 1

P Setup a shared folder using the virtualbox GUI, and call it workvm, and then

sudo mount -t vboxsf -o uid=1000,gid=1000 workvm /home/ubuntu/code/workvm

This will mount the folder setup for sharing in the previous step, inside guest at /home/ubuntu/code/workvm.

Dark New Tab Page in Firefox Link: https://superuser.com/questions/1288920/how-to-modify-the-new-tab-page-in-firefox-with-stylish

November 9, 2018

ElasticSearch – Delete Everything and Start from Scratch

curl -X DELETE 'http://localhost:9200/_all'

Managing WiFi

Encrypted Arch Install

Note the --``max_allowed_packet parameter which might be required sometime.

Running phpmyadmin against an arbitrary remote server

First, make sure remote access is enabled on remote (it requires 2 things): https://stackoverflow.com/questions/14779104/how-to-allow-remote-connection-to-mysql

Then launch the docker container with PMA_ARBITRARY set to 1:

docker run --name myadmin -d -e PMA_ARBITRARY=1 -p 8080:80 phpmyadmin/phpmyadmin

Ubuntu Kiosk https://tutorials.ubuntu.com/tutorial/secure-ubuntu-kiosk#0

Untitled

January 4, 2019

One Hot Encoding

January 22, 2019

Web Crawling

February 23, 2019

Bulk Rename Files on CLI

Concatenate MP3 Files

ffmpeg -i "concat:01.mp3|02.mp3|03.mp3|04.mp3|05.mp3|06.mp3|07.mp3|08.mp3|09.mp3|10.mp3|11.mp3|12.mp3" -acodec copy outlander_s1.mp3

February 26, 2019

Docker Inside Docker

First, find the GID of the docker group on host

# ls -alth /var/run/docker.sock 
srw-rw---- 1 root 974 0 Feb 24 16:37 /var/run/docker.sock

It is 974. So we add a group docker in the container with GID 974 and then modify the deepak user to add them to the docker group.

groupadd -g 974 docker
usermod -a -G docker deepak

GPG Sign Inside Docker Got an error inside Docker when trying to use gpg: gpg: signing failed: Inappropriate ioctl for device.

From: https://github.com/keybase/keybase-issues/issues/2798

export GPG_TTY=$(tty)
echo "test" | gpg --clearsign # Works

Using GPG Inside Docker Container: https://nixaid.com/using-gpg-inside-a-docker-container/

March 18, 2019

Python Cheatsheet: https://gto76.github.io/python-cheatsheet/

May 24, 2019

Docker MySQL server

Starting a docker based MySQL server w/ a persistent data folder:

Step 1: Create the data directory (make sure it is empty)

mkdir joblang-mysql

Step 2: Start docker and point it to the data directory

docker run --name joblang-mysql -v ~/joblang-mysql/:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=password   -p 3306:3306 -d mysql:5.7.26

The command below also sets the root password to password and exposes port 3306 inside the container to host

Step 3: Try to connect to server from the host:

mycli -h localhost -u root

May 31, 2019

Managing dotfiles using a bare git repo https://www.atlassian.com/git/tutorials/dotfiles

June 6, 2019

Counting no. of rows in a dynamodb database

aws dynamodb scan --table-name name_gender_mapping --select "COUNT" --region us-east-2

June 21, 2019

Packaging **pipenv** applicatons for AWS Lambda

SITE_PACKAGES=$(pipenv --venv)/lib/python3.7/site-packages
DIR=$(pwd)

# Make sure pipenv is good to go
pipenv install

cd $SITE_PACKAGES
zip -r9 $DIR/OldPosts.zip *

cd $DIR
zip -g OldPosts.zip *.py#6

June 26, 2019

pytest tips and tricks

July 1, 2019

Install xml2 R package on OS X Mojave

brew install libxml2

However, after this I cannot seem to get pkg-config to detect the libxml2.pc file. So the R package installation is failing. I was able to get it to install by explicitly specifying the install paths and doing:

install.packages("xml2", configure.vars='INCLUDE_DIR=/usr/local/opt/libxml2/include/libxml2 LIB_DIR=/usr/local/opt/libxml2/lib/')

Automatically sending 200 response in nginx

server {
  location /elb-status {
    access_log off;
    return 200;
  }
}

Serverless Infrastructure as code config using Terraform

fzf quick guide

https://www.youtube.com/watch?v=qgG5Jhi_Els

July 19, 2019

Querying dynamodb Simple Example:

import boto3
from boto3.dynamodb.conditions import Key, Attr

dynamodb = boto3.resource("dynamodb", region_name="us-east-2")
company_reviews = dynamodb.Table("company_reviews")

res = table.query(P 
                  FilterExpression=Attr('source').eq("indeed"),
                  ProjectionExpression='review_id')

July 31, 2019

Finding fast mirrors in Arch

curl -s "https://www.archlinux.org/mirrorlist/?country=US&protocol=https&use_mirror_status=on" | sed -e 's/^#Server/Server/' -e '/^#/d'| rankmirrors -n 5 -

August 13, 2019

terraform variables

variable "user_names" {
  description = "Create IAM users with these names"
  type        = list(string)
  default     = ["neo", "trinity", "morpheus"]
}

resource "aws_iam_user" "example" {
  count = length(var.user_names)
  name  = var.user_names[count.index]
}

# Alternate (better version)
resource "aws_iam_user" "example" {
  for_each = toset(var.user_names)
  name     = each.value
}

output "all_arns" {
  value       = aws_iam_user.example[*].arn
  description = "The ARNs for all users"
}

August 19, 2019

Quote

The hardest parts of SWE are consensus building, weighing multiple plausible alternatives, risk estimation, team coordination, expectation management, and so many other things that are not “coding”. One of the best SWEs I’ve known used to say “coding is easy and boring”.

August 20, 2019

DynamoDB Python Deserialization

from boto3.dynamodb.types import TypeDeserializer

deser = TypeDeserializer()

...
<in handler>
    for record in event['Records']:
        old = record['dynamodb'].get('OldImage')
        new = record['dynamodb'].get('NewImage')
        if old:
            d = {}
            for key in old:
                d[key] = deser.deserialize(old[key])

August 23, 2019

Vagrant resize hard disk

https://nmrony.info/change-disk-size-of-a-vagrant-box/

MySQL remote host access

GRANT ALL ON *.* to user@'%' IDENTIFIED BY 'password';

September 13, 2019

Package Management in Go Simple cheatsheet of all the possible commands that one might need for package management in Go.

https://deepsource.io/blog/go-modules

Dec 12, 2019

MySQL view collations

SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';

Oct 16, 2019

Using an SSH Tunnel to access an HTTP(S) server inside a private network in AWS:

ssh -N DevBoxRandD -L8080:5pw91ji32f.execute-api.us-west-2.amazonaws.com:443
$ curl -H "Host: 5pw91ji32f.execute-api.us-west-2.amazonaws.com" --insecure https://localhost:8080/prod/gender?name=Amit

{"sources": ["FBP", "Global Name Tracker", "SSA", "US Census", "Ultimate Names"], "prob": 0.9012345679012346, "name": "amit", "gender": "m"}

Oct 14, 2019

Here is an example of a query in MySQL which can approximate the LAST aggregate function that is provided by some other databases

select x.employer_name_clean, x.c, h.EMPLOYER_ADDRESS, h.EMPLOYER_POSTAL_CODE 
FROM
  (
     select employer_name_clean, MAX(id) as maxid, count(employer_name_clean) as c 
     from h1b_data_cleaned 
     group by employer_name_clean
     having c > 5 
     order by c desc
  ) as x 
INNER JOIN h1b_data_cleaned as h on x.maxid=h.id

can i use aggregation function (LAST) in mysql?

Oct 07, 2019

SQL query to add an autoincrement column to a table:

ALTER TABLE `users` ADD `id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY

Oct 03, 2019

Go Modules and Private Git Repos

Need to do some git config trickery to get private repos to work with Golang:

Go Modules with Private Git Repositories

Sep 20, 2019

Using SSM and Secrets Manager in AWS to store config

Sharing Secrets with AWS Lambda Using AWS Systems Manager Parameter Store | Amazon Web Services

Apart from other useful info, this post contains an interesting pattern to initialize an app the first time the lambda handler is called, and reuse it for subsequent invocations. This is useful for alias-specific configuration.

def lambda_handler(event, context):
    global app
    # Initialize app if it doesn't yet exist
    if app is None:
        print("Loading config and creating new MyApp...")
        config = load_config(full_config_path)
        app = MyApp(config)

    return "MyApp config is " + str(app.get_config()._sections)

Find size of tables

Mysql list tables and sizes - order by size

SELECT TABLE_NAME, table_rows, data_length, index_length, 
round(((data_length + index_length) / 1024 / 1024),2) "Size in MB"
FROM information_schema.TABLES WHERE table_schema = "schema_name"
ORDER BY (data_length + index_length) DESC;

Replace schema_name with the name of database.

Apr 13, 2020

Setting up IntelliJ on Ubuntu

sudo apt-get install openjdk-11-jdk openjdk-11-source

This will install the source files under /usr/lib/jvm/openjdk-11/lib/src.zip.

After that navigate to FileProject StructurePlatform SettingsSDKs, and add a Sourcepath that points to the file above.