Download a list of files using a shell (bash) script.
Today I'm going to share something interesting which will help you to download a large number of files at once. For one of my projects, I had to find images from Google, and upload them one by one to a particular website and create posts. At first, this was not much difficult as the number of images I needed was very little. But it kept growing and growing and I had to manually do the same thing repeatedly. Since I'm a lazy person by nature, I wanted to automate the process and here is a part of it, which I did to download the images quickly.
Unfortunately I couldn't automate the Googling part because it needed some human eyes to find quality images (I didn't want to go to the level of automation using computer vision ;) ). So however I searched for images, then whenever I found an interesting image, I right clicked on it and opened the image in a new tab (using ctrl + click). I kept doing this until I had enough images.
Here's the interesting part:
I installed this awesome Firefox extension in my browser, which allowed me to copy all of the URLS which were currently open in my brower. That was very convenient given that I had to copy each URL one by one when I began this task.
So now I have a list of URLS of the images I wanted to download. Now what? I pasted the copied list to a text document and opened another text document to write my shell (bash) script. So here it is.
#!/bin/bash
file="urls.txt"
while IFS= read line
do
echo "$line"
wget "$line"
done <"$file"
Explanation: First I assign the name of the file I want to read into the file variable. Then I simply read the file line by line and run a wget command with the url to download the file. To run this script, all I had to do was fire up a terminal, grant the execute permissions to the file and run it using the following command.
$ ./download.sh
download.sh is the name of your shell script.
That's it folks. I hope this will help someone :) Please leave a comment if you think this helped you or if you think I miss anything.