The curl command can be used to transfer files over a network or from the internet using a number of different formats including http, https, ftp and even smb.
The curl command has lots of different authentication methods. You can use it to access FTP sites, send emails, connect to SAMBA addresses, upload and download files and many other things.
- Feb 13, 2014 - The powerful curl command line tool can be used to download files from just about any remote server. Longtime command line users know this.
- Downloading file from FTP using cURL. I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer.
The curl command has a lot of parameters to learn. You can see a full listing of them at the curl manual page.
Basic curl Command Usage
The curl command can be used to download files from the internet, but in its basic form, you can download the web page content straight to the terminal window.
CURL stands for command Line URL and is a simple, yet powerful, command line utility that gives the ability to download content using a lightweight executable that provides cross-platform support. CURL is community supported and is often a packaged part of some *nix systems already. Download a sequence of files with curl Posted in Linux/Unix/BSD - Last updated Apr. In the past to download a sequence of files (e.g named blue00.png to blue09.png) I've used a for loop for wget but there's a simpler and more powerful way to do the same thing with curl.
For example, enter the following command into a terminal window:
The output will scroll up in the terminal window and it will show you the code for the linked webpage.
Obviously, the page scrolls too fast to read and so if you wish to slow it down you should use either the less command or the more command.
Output the Contents of curl to a File
The problem with the basic curl command usage is that the text scrolls very fast. If you are downloading a file such as an ISO image, you don't want this going to the standard output.
To save the content to a file, all you have to do is specify the minus o (-o) switch as follows:
To download the page linked to in the basic command usage section, all you have to do is enter the following command:
After the file has downloaded, you can open it in an editor or its default program determined by the file type.
You can simplify this further by using the minus O switch (-O) as follows:
This will use the filename portion of the URL and make it the filename that the URL is saved to. In the above instance, the file will be called curl.htm.
Run the Curl Command In the Background
By default, the curl command shows a progress bar telling you how much of the transfer is left and how much data has been transferred.
If you just want the command to run without a progress bar, you'll need to run it in silent mode. Then run it as a background command.
To run a command silently use the following command:
To get the command to run in the background, you need to use the ampersand (&) as follows:
Running the culr command without the progress bar is more streamlined, but you won't know how long the transfer will take or the estimated time it should complete.
Downloading Multiple URLs with Curl
You can download from multiple URLs using a single curl command.
In its simplest form, you can download multiple URLs as follows:
Imagine you have a folder with 100 images all called image1.jpg, image2.jpg, image3.jpg, etc. You wouldn't want to have to type in all of these URLs.
To do this, you can use square brackets to supply a range. For example, to get files 1 to 100, you can specify the following:
You can also use curly brackets to specify multiple sites with similar formats.
For example, imagine you want to download www.google.com and www.bing.com. You can simply use the following command to accomplish this:
Displaying Progress
By default the curl command returns the following information as it downloads a URL:
- Total %: Percent of total transfer completed.
- Total bytes: Total size of the transfer in bytes.
- Received/Transferred %: Percentage of transfer completed.
- Received/Transferred bytes: Number of bytes downloaded.
- Average download speed: Download speed in bytes per second.
- Average upload speed: Upload speed in bytes per second.
- Total time: Estimated time to complete the current operation.
- Time spent: Time passed since the start of transfer.
- Time left: Expected time to complete the transfer.
- Current speed: Current transfer speed in bytes per second.
If you would prefer a simple progress bar, just specify the minus hash (-#) switch as follows:
Handling Redirects
Imagine you have specified a URL as part of the curl command and think you have the right address to download a large file only to come back later to find that all you have is a webpage stating 'this page has been redirected to www.blah.com'. That would be annoying, wouldn't it?
The curl command is clever in that it can follow redirects. All you have to do is use the minus L switch (-L) as follows:
How To Use Curl To Download A File
Reduce the Download Rate
If you are downloading a large file and you have a poor internet connection then you might annoy the family if they are trying to do stuff on the internet as well.
Fortunately, you can reduce the download rate with the curl command.
The rate can be specified in kilobytes (k or K), megabytes (m or m) or gigabytes (g or G).
Keep in mind that reducing the download speed will increase the time it takes for a transfer to complete.
Download Files From an FTP Server
The curl command can handle more than just HTTP file transfers. It can handle FTP, GOPHER, SMB, HTTPS and many other formats.
To download files from an FTP server use the following command:
If you specify the name of a file as part of the URL then it will download the file but if you specify the name of a folder it will return a folder listing.
You can also use curl to upload files to an FTP server by using the following command:
The filenames and <URL> can use the same pattern matching as for downloading multiple HTTP files.
Passing Form Data to a Form
You can use curl to fill in an online form and submit the data as if you had filled it in yourself. Many popular services such as Google block this sort of usage.
Imagine there is a form with a name and email address. You can submit this information as follows:
Html Code To Download A File
There are various ways of transferring form information. The above command uses basic text, but if you want to use multi encoding which allows image transfer then you will need to use the minus F switch (-F).
Learn how to download any file using command line from internet or FTP servers to your Linux server. Get files in your server in seconds!
There are many times when you want a file on your Linux server from Internet or FTP server and you are on command line terminal! When using GUI of Linux, its easy to get file by using browsers but for command line its little bit difficult.
We have 4 tools here to help you with the task! They are :
- wget
- curl
- elinks
- w3m
wget :
Most popular utility! wget is a package you can install and use it right out of the box. You can install it with YUM or APT package. Once install you can use it with supplying URL of the targeted download.
2 4 6 8 10 12 | # wget https://kerneltalks.com/image.png --2017-03-0506:56:54--http://kerneltalks.com/image.png Connecting tokerneltalks.com|208.91.198.91|:80...connected. Length:12477(12K)[image/png] 100%[=>]12,477--.-K/sin0s 2017-03-0506:56:55(782MB/s)-“image.png”saved[12477/12477] |
In above example we have downloaded one picture file from internet! File will be saved in your present working directory by default.
2 4 6 8 10 12 14 16 | # wget ftp://rpmfind.net/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/s/systemd-233-2.fc27.x86_64.rpm --2017-03-0506:58:54--ftp://rpmfind.net/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/s/systemd-233-2.fc27.x86_64.rpm Resolving rpmfind.net...195.220.108.108 Connecting torpmfind.net|195.220.108.108|:21...connected. >SYST...done.>PWD...done. >TYPEI...done.>CWD(1)/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/s...done. >PASV...done.>RETR systemd-233-2.fc27.x86_64.rpm...done. 100%[=>]3,179,4961.85M/sin1.6s 2017-03-0506:58:57(1.85MB/s)-“systemd-233-2.fc27.x86_64.rpm.1”saved[3179496] |
In this example we used wget to download file from FTP server. It used anaonymous login to get into server and download file!
There are several options which you can used according to your requirement. Listing below few important ones.
- -b : send copy progress in background
- -c : continue download (breaked or paused donwload resume)
- -r : recursive (download all files in destination)
- -A fileextension : download only files with specified extension
curl :
Curl is simple downloader which supports many protocols for file transfer few being ftp, https, https, telnet etc. It can be installed using same above method yum install curl or apt-get install curl.
Curl renders file downloaded to its best known way. Like if you try to download html URl then it will render it and shows you html code on terminal :
2 4 | # curl https://kerneltalks.com <!DOCTYPE html><html lang='en-US'prefix='og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#'><head><meta charset='UTF-8'/><title>Kernel Talks-Unix,Linux&scripts.</title><meta name='viewport'content='width=device-width, initial-scale=1'/><meta name='google-site-verification'content='jeFc7PXM8ZxDY5awb8nCCD5-bYwj5S7bwsAIgp1JIgU'/><meta name='msvalidate.01'content='920806CD9A79B08EC8477C0D440658A4'/><meta name='p:domain_verify'content='738d0b16e329ab01cc894a68d2adda34'/><meta name='yandex-verification'content='bd079834c4df4ebf'/> |
See above example where it shows html code of URL. Same way if you get text file it will show you text file content on terminal.
To only download file without trying to read/open it on terminal use option -o with curl.
2 4 6 | # curl -O ftp://rpmfind.net/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/s/systemd-233-2.fc27.x86_64.rpm %Total%Received%Xferd Average Speed TimeTimeTimeCurrent 1003104k1003104k00361k00:00:080:00:08--:--:--618k |
It will download file and progress will be shown on terminal in real time.
elinks :
elinks is text beased browser which supports colors, rendering, tabbed menus etc. Mostly its preloaded with Installations but if not you can install it using yum or apt-get. Let try to download website using elinks http://kerneltlaks.com :
Above example shows elinks renders website in text mode (kind of) on terminal!
If you try to download image (or any type of) file it will show you below GUI screen (within terminal) with options to choose from what to do next. If you choose save then it will download file and keep it.
w3m :
Last tool of this article to download internet based files is w3m. w3m is text based www (world wide web) client. Installation steps remain same yum/apt-get install w3m
It also opens up text mode GUI screen like elinks and gives you interactive options to choose actions. w3m ftp://rpmf…../…86_64.rpm opens :
If you right click on terminal window (normally we dont!!) it does show you menu you can use to perform various actions.
You can navigate through this menu using keyboard arrow keys or even using mouse clicks. You can even use short cut keys defined for each menu item in brackets beside them.
All user interactive commands/options are shown in lower left corner of the terminal and choices can be submitted there only.
In all if you are looking for a simple tool, less eye rolling on screen, fast way to get file on server then wget is the choice you should make IMHO! Let me know which command line tool you use for downloading internet files in comments.