Bash - Switch STDERR to STDOUT


Bash Switch STDERR to STDOUT

Today I wrote a bash script to create a very simple porcelain client on Git to add and commit files without entering all the git add, commit commands. I personally don't like to use Git clients with GUIs because they are somewhat complex, and I really love working on the terminal and the comfort of typing on my keyboard. But typing all the commands every time when working with Git isn't productive as well. So I decided to create a small client so I can just run it and do all the basic repetitive tasks through it.

Since I wanted to run it in the terminal and also to reduce the usage of mouse, I used whiptail to generate all the UIs inside the terminal to select files and commit. However, while working with it, I noticed that some components of whiptail, like checklist, for example, uses STDERR to return its output. I have no idea why it prints the output on STRERR, but I wanted it to print to the STDOUT. So here's how to do that redirection.

You just have to add the following at the end of the line you need.

3>&1 1>&2 2>&3
Example:

whiptail --separate-output --title "Select files to add" --checklist " " 10 60 5 '$filelist' 3>&1 1>&2 2>&3

What it does?

Well, I hope you already know what are file descriptors. If not, basically there are three of them,

0 - STDIN
1 - STDOUT
2 - STDERR

in addition to these specific FDs, there are 3-9, and they can be created as we want. In the above line, what we actually do is,

3>&1 = we create a new file descriptor 3, and points it to the STDOUT (& sign is used to refer to another FD, and here we actually create a temporary FD 3 to hold the STDOUT).

1>&2 = Then we redirect the STDOUT to STDERR. We actually needed the above line because we switch it here, and if not for the above, we would lose the reference to STDOUT after the switch.

2>&3 = Now we simply redirect the STDERR to the FD 3 we created, which is currently redirected to the STDOUT.

Now you see that we have switched STDERR to STDOUT and vice versa :) I hope this is clear and will be helpful to someone. I only wanted to write this here so I can refer to it on a later day in case I forget. But if you ever come across this post, don't forget to leave a comment.

Ubuntu touch: Change the created or modified date of a file using terminal.

Ubuntu touch: Change the created, access or modified date of a file using terminal.


Last week I wrote an automation script, which (a part of it) generated some files, and I wanted to store them for a maximum of three days. So along with the script, I wrote another bash script to delete any files which are older than three days, and I set it to run as a cron job.

After writing all the scripts, I wanted to test whether they work. For tests, I needed some files which are at least three days old and useless at the same time because my script will be deleting them.

So how did I do that?

I believe you all know about the touch command. What does it actually do? Well, I'm sure most of us will say "we use it to create a file". It's true. But that's not what touch command does. Its real purpose is to change the access and modification time of a file. It will create a new file only if it doesn't exist ;) So this blog post is about how to do it. I thought of sharing it in my blog for my own reference as well as a help for those who will be looking for it in the future ;)

In the touch's manual page, it says:

Update  the  access  and modification times of each FILE to the current time.

A FILE argument that does not exist is created empty, unless -c  or  -h
is supplied.

A  FILE  argument  string of - is handled specially and causes touch to
change the times of the file associated with standard output.

Mandatory arguments to long options are  mandatory  for  short  options
too.

So, that's it. You just have to add the -d parameter and pass in a timestamp and it will create the file with given date / time. If the file already exists, it will modify the access / created time to the given one. Too easy right? For my specific requirement, the following command was enough for me.

touch -d '4 day ago' test.txt

As you can see, the value given to the -d parameter could be a human friendly syntax like shown above. You can also pass in a timestamp if you wish. Please read the manual page for more information on that.

I hope this will be helpful for someone :)

View and edit files inside a docker container.

How to view and edit files inside a docker container.


Today I deployed a new service in a docker container and found that service wasn't working as expected. Since it was working in my local environment, I guessed that it could be due to a small issue. So I decided to access the files inside the docker container and do some manual debugging. I wanted to write down the command to access the terminal of the container in my blog for my own reference. Hope this will help others too.

So here's how to get into the terminal / bash prompt of your container. Just run the following command.

docker exec -it <your-container-name> bash
That's it. I hope you know that you can get a list of all running instances by giving the docker ps command ;)

One last thing, since I mentioned about editing files. Docker by default, doesn't come with any editors like nano or vim. You may need to install them manually. It's super easy, in case you don't remember, just run:

apt update
apt install vim
Hope it helps :)

Postgresql cannot start and no errors in logs - Fix


I have been doing a lot of work with Postgresql recently including replication, clustering, automatic failovers and so on. During one of my tests, I came through this situation where I tried to restart Postgresql server and it shows the server is started, but if I check the status of it, it is always shown as down, and I was unable to connect to it.

After doing everything I could, I felt this could be something related to the start up process, and remembered I disabled the starting of the Postgresql server while setting up the pacemaker cluster. So that was it. I just changed "disabled" to "auto" and it started the service :)

Just run the following.

echo auto > /etc/postgresql/9.6/main/start.conf

and start the server.

service postgresql start

One other thing you could try is, see if you have enough space left on your disks. That could also be a reason for Postgresql to not start, although it wasn't the case with me.

Check the disk space with following command.

df -h

I hope this will be useful for someone :)

systemctl disable alternative command for Ubuntu 14.04



systemctl disable alternative command for Ubuntu 14.04


If you are using Ubuntu 15 or any higher version, you can use the systemctl command to enable or disable services. But before Ubuntu 15, it is not available.

So here is an alternate way to do the same in earlier versions like Ubuntu 14.04. I recently wanted the same and thought of writing it down here for my own reference as well as a help for those who looking for it.

Enter the following command along with the service name you want.

update-rc.d -f <service> remove
You can use the following command to get a list of services running in your server.

service --status-all
I hope this will be useful to someone :) Please don't forget to leave a comment if you know any better way.

Fix - Ubuntu hangs at the end of file copying

Fix - Ubuntu hangs at the end of file copying


I had this problem with Ubuntu for a long time and I've been looking for a solution that works. After trying out many different things, I finally found something that really worked and I thought I should share it in my blog for my future reference as well as a help for those who looking for it.

If your running a x64 bit Ubuntu or other Linux and find USB transfers hang at the end apply this fix:

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes


I suggest you edit your /etc/rc.local file to make this change persistant across reboots.

sudo nano /etc/rc.local

Go to the bottom of the file and leave a space then paste in those two lines.

Save the file with ctrl + x then press y.

To revert the changes enter this in console and remove the lines in /etc/rc.local

echo 0 > /proc/sys/vm/dirty_background_bytes
echo 0 > /proc/sys/vm/dirty_bytes


More info and references: https://unix.stackexchange.com/questions/107703/why-is-my-pc-freezing-while-im-copying-a-file-to-a-pendrive/107722#107722


Source: https://gist.github.com/2E0PGS/f63544f8abe69acc5caaa54f56efe52f

I directly copied it from the above github gist.

Honestly I don't have any idea about the properties which are set above, and I didn't even do any research on them. However it seems working and shows the real copying progess, and doesn't hang at the end of the copying.

Easy setup Streaming Replication for Postgres 9.6 in Ubuntu


Recently I installed Postgresql 9.6 on two Ubuntu servers and configured streaming replication between them. Although there are many articles and guides covering this topic, I noticed that most of them are missing some important steps or they contained wrong information. After the setup was done, I thought I should write it down as a complete, simple guide in my blog for future readers.

Here's the most basic and the simplest way to setup streaming replication in Postgresql 9.6.

On your Master Server
==================

1. First we need to create a user for replication.
su - postgres
psql
CREATE USER replication REPLICATION LOGIN ENCRYPTED PASSWORD '123';

You can list the users with the following command:
\du
Edit the /etc/postgresql/9.6/main/postgresql.conf file and do the following changes.

listen_addresses = '*'
wal_level = replica
max_wal_senders = 5
wal_keep_segments = 32
archive_mode = on
archive_command = 'test ! -f /var/lib/postgresql/9.6/main/archive/%f && cp %p /var/lib/postgresql/9.6/main/archive/%f''

Now we need to edit the pg_hba.conf file and add a record for the standy server so that it can connect to the primary.

host replication replication 192.168.3.169/32 md5
Note: 192.168.3.169 is the IP address of standby server.

Now create a new directory inside of the 'main' directory for the archive configuration - run the below command as postgres user:

mkdir -p /var/lib/postgresql/9.6/main/archive/

On your Standby Server
===================

service postgresql stop
We shall keep a backup of the postgres data directory.

mv /var/lib/postgresql/9.6/main /var/lib/postgresql/9.6/main_backup
Now we should migrate the data dir from the master server using the pg_basebackup tool.

pg_basebackup -h 192.168.4.168 -D /var/lib/postgresql/9.6/main -U replication -v -P
Note: 192.168.4.168 is the IP address of the Master server.

After the base backup is complete, we should create a recovery.conf file.

/var/lib/postgresql/9.6/main/recovery.conf

standby_mode = 'on'
primary_conninfo = 'host=192.168.4.168 port=5432 user=replication password=123'
restore_command = 'cp /var/lib/postgresql/9.6/main/archive/%f %p'
trigger_file = '/tmp/postgresql.trigger'

Switch back to the root user.

exit
Start the Postgres service

service postgresql start
Verification
==========

Now we have finished setting up streaming replication between the two servers. To verify that both servers are working as expected, switch to the postgres user and run the following command in the Master Server.

su - postgres
psql -x -c "select * from pg_stat_replication;"
You should see the client_address property pointed to standy server's IP address and the state as streaming.

Note: I took the above restore and archive commands from the Postgresql documentation here: https://www.postgresql.org/docs/9.6/continuous-archiving.html

That's it. I tried my best to keep it as simple as possible. This is only a basic setup to give you an idea on how to setup streaming replication. For a production environment, you might need to set proper values depending on the requirements.

I hope this post will be helpful for someone :) Don't forget to leave a comment if it helped.

FATAL: the database system is starting up - Postgres Streaming Replication

FATAL: the database system is starting up - Postgres Streaming Replication


I was setting up streaming replication between two Postgresql servers in active passive mode. After the setup was done, I tried to connect to the slave node to see if everything works fine and received the following error:

FATAL:  the database system is starting up

I was thinking for a while before I realized that I didn't set the slave node to hot_standby mode. It seems like a common mistake everyone does, so I thought of writing it down here hoping someone would find it useful.

If you get this error, this is what you have to do:

Open your postgresql.conf file.

vim /etc/postgresql/9.6/main/postgresql.conf
Locate the following

hot_standby = off
and change it as follows:

hot_standby = on

Save the file, then:

service postgresql restart

Now you can query your slave server ;)

Fix gpg: no valid OpenPGP data found error in Ubuntu



Fix gpg: no valid OpenPGP data found error in Ubuntu

Today I was installing Postgres 9.6 in Ubuntu 14 server and while adding the key, I got this error.

gpg: no valid OpenPGP data found error

This is the command I entered:

wget -q -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -

I tried several solutions and found a way to get past this error. To solve, all you have to do is download the key separately and add it.

Here we basically the run the above in two seperate commands. This is how to do it.

First get the key.

wget https://www.postgresql.org/media/keys/ACCC4CF8.asc

Then add it:

sudo apt-key add ACCC4CF8.asc

That's it. Hope it will help someone ;)

Fix tmux: error while loading shared libraries: libevent-2.0.so - upgrade to Ubuntu 18.04


Fix tmux: error while loading shared libraries: libevent-2.0.so - upgrade to Ubuntu 18.04

I have been using Tmux for some time with Ubuntu 16.04 and it was all working so well until I upgraded my system to Ubuntu 18.04. When I tried to run tmux via the command, it interrupted with an error as shown below:

tmux: error while loading shared libraries: libevent-2.0.so

So I checked a little bit deeper and found the solution to this. It seems like tmux in Ubuntu 18.04 requires the shown version of libevent and since I installed tmux under Ubuntu 16.04, I'm missing it. That's pretty obvious.

You should also note that Tmux comes with all the required libraries in Ubuntu repositories if you use Ubuntu 18.04. So in order to fix it, all you have to do is remove the version of tmux you already have and reinstall it.

First let's remove tmux.

sudo apt purge tmux*
This will remove all tmux related packages from your system. Now if you enter the command tmux in your terminal, you should get a message saying it's not installed. If it still shows the same error, then you have to manually remove tmux.

To see which tmux binary is used in your system, run the following command.

which tmux

It will display the path to tmux. You can remove it manually by running:

sudo rm $(which tmux)

Fine, now tmux is removed from your system. Now let's install it using apt.

sudo apt install tmux

After installing, you can start tmux by entering the tmux command in your terminal. If you get some error saying tmux is not found, then its probably because your system is still using the old path to tmux, which we deleted before.

To verify that, enter the following command.

type tmux

If you get a response something to similar to this :

tmux is hashed (/usr/local/bin/tmux)

Then it means you have to clear the old hash, so it will identify the new tmux path.

You can remove the hash by running the following command.

hash -d tmux

This will remove the hash entry and now if you again try to run tmux, it should work as expected.

I hope this will help somebody :) Feel free to comment and ask any questions.