Download a list of files using a shell (bash) script.

Download a list of files using a shell (bash) script.


Today I'm going to share something interesting which will help you to download a large number of files at once. For one of my projects, I had to find images from Google, and upload them one by one to a particular website and create posts. At first, this was not much difficult as the number of images I needed was very little. But it kept growing and growing and I had to manually do the same thing repeatedly. Since I'm a lazy person by nature, I wanted to automate the process and here is a part of it, which I did to download the images quickly.

Unfortunately I couldn't automate the Googling part because it needed some human eyes to find quality images (I didn't want to go to the level of automation using computer vision ;) ). So however I searched for images, then whenever I found an interesting image, I right clicked on it and opened the image in a new tab (using ctrl + click). I kept doing this until I had enough images.

Here's the interesting part:

I installed this awesome Firefox extension in my browser, which allowed me to copy all of the URLS which were currently open in my brower. That was very convenient given that I had to copy each URL one by one when I began this task.

So now I have a list of URLS of the images I wanted to download. Now what? I pasted the copied list to a text document and opened another text document to write my shell (bash) script. So here it is.

#!/bin/bash

file="urls.txt"

while IFS= read line
do
    echo "$line"
    wget "$line"   
done <"$file"

Explanation: First I assign the name of the file I want to read into the file variable. Then I simply read the file line by line and run a wget command with the url to download the file.

To run this script, all I had to do was fire up a terminal, grant the execute permissions to the file and run it using the following command.

$ ./download.sh

download.sh is the name of your shell script.

That's it folks. I hope this will help someone :) Please leave a comment if you think this helped you or if you think I miss anything.

Solved - How to exit a telnet session?

How to exit a telnet session?
How to exit a telnet session?
It's funny, but I was fumbling with the terminal to exit a telnet session today. I rarely have to use telnet so I'm not much experienced with it. But if someone ever want to exit from a telnet session this is what you have to do.

Basically when you start a telnet session, this is what you would see most of the times:

~$ telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.


Now what? Pressing Ctrl + c, z, x, d will not do anything. I was too lazy that I didn't pay much attention to what was printed in the console. If I had read that, I would have easily exit the session.

It says:  Escape character is '^]'.

So that's it. You need to press Ctrl + ] keys and it will take you back to the telnet prompt where you can either press Ctrl + D or enter the command "quit" and press enter.

~$ telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.


One thing to note is that I use Terminator as my terminal emulator and this method doesn't work in it. I Still can't exit from that (: But this definitely works in the gnome terminal.

I hope this will help someone :) Please don't forget to leave a comment if you ever come across this post. Feel free to correct me or add any methods you know other than this.

Fix libQt5Charts.so.5: cannot open shared object file error with Redis Desktop Manager in Ubuntu



A few days ago, I was looking for a Redis client for Ubuntu and found that there is one called Redis Desktop Manager. Although it's free and open source, it only provides builds for Windows. Ubuntu and Mac OS users have to subscribe by making a monthly payment or they can build it by themselves using the source (: So I opted for the second option and built and installed it in my Ubuntu box since it was for a small task.

Then today, when I tried to run the RDM again, I got this error and I was wondering what might have changed in the system. In fact there was nothing changed so I had to dig a little bit more into the error and found why I got this error. So here's how I fixed it.

If you haven't built the Redis Desktop Manager (RDM) by source before, this is how to do it. (It's all well documented)

~$ git clone --recursive https://github.com/uglide/RedisDesktopManager.git -b 0.9 rdm && cd ./rdm

~$ cd src/
~$ ./configure
~$ source /opt/qt59/bin/qt59-env.sh && qmake && make && sudo make install
~$ cd /usr/share/redis-desktop-manager/bin
~$ sudo mv qt.conf qt.backup

Now it's done and you can simply run RDM by entering the command:

~$ cd /usr/share/redis-desktop-manager/bin/
~$ ./rdm

or simply by using:

~$ sh /usr/share/redis-desktop-manager/bin/rdm.sh
I believe that you are reading this blog post means you got either of the following errors when you run the above command:

/usr/share/redis-desktop-manager/bin/rdm.sh: 2: /usr/share/redis-desktop-manager/bin/rdm.sh: source: not found
/usr/share/redis-desktop-manager/bin/rdm: error while loading shared libraries: libQt5Charts.so.5: cannot open shared object file: No such file or directory

or just:

./rdm: error while loading shared libraries: libQt5Charts.so.5: cannot open shared object file: No such file or directory
It says it's unable to load the libQt5Charts.so library. So if you are sure that you have installed (I bet you have) and still get this error, you need to do a few things before running RDM.

First you need to tell it where you have the QT libraries. When you build from the source, the QT libraries are installed at /opt/qt59 directory.

So just enter the command

~$ source /opt/qt59/bin/qt59-env.sh
Now if try to run RDM again, it should work fine :) Since this seems reduntant for us to enter this command everytime we want to run the Redis Desktop Manager, you can edit the rdm.sh file and add the above line on top of it.

Note: If you paid enough attention to the build process, you may have noticed that we ran the above command during building it from the source. That's the reason why it worked a few days ago when I installed it.

So let's update /usr/share/redis-desktop-manager/bin/rdm.sh file.

~$ vim /usr/share/redis-desktop-manager/bin/rdm.sh
#!/bin/bash
source /opt/qt59/bin/qt59-env.sh
DIR=$(dirname "$(readlink -f "$0")")
export LD_LIBRARY_PATH="$DIR/../lib":$LD_LIBRARY_PATH
$DIR/rdm

Notice that we added the line: source /opt/qt59/bin/qt59-env.sh after the shebang. Now save it by pressing ESC then :wq in vim and RDM should load fine :)

I also found that having to enter long paths to run RDM is overkill. So I just added a softlink to my /usr/bin directory using the below command.

ln -s /usr/share/redis-desktop-manager/bin/rdm.sh /usr/bin/redis-desktop
Now I can run Redis Desktop Manager using the redis-desktop command.

That's it folks. I hope this will be helpful to someone. Please don't forget to let me know if I have missed anything, and leave a comment if you know a better way to do this.