Downloading Multiple Files with Curl Simultaneously

Wouldn't it be great if you could use php and curl to download multiple files simultaneously using built-in curl functions? You can!

---

Example Code

Note that this won't work out of the box or anything, I was passing this function an array containing a bunch of mod_rewrite variables, but you can get an idea of how to use the code.

function curlit_rewritecond($U,$pass){
    global $RPASS,$RSITE;
	$RPASS=$pass;

    $FF_HDR=array("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
	"Accept-Language: en-us,en;q=0.9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0.6,nl;q=0.5,fr;q=0.5,it;q=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0.1",
	"Accept-Encoding: gzip,deflate","Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7","Keep-Alive: 300","Connection: keep-alive","Pragma:");

		if($fp = tmpfile()){
        $mh = curl_multi_init();
        foreach ($U as $i => $ur) {
			$url=$RSITE.'?Q='.$ur;
            if (!$url_info = parse_url($url)) die('bad url '.$url);
            $ch[$i] = curl_init($url);
            curl_setopt ($ch[$i], CURLOPT_HEADERFUNCTION, 'aacurlheader');
            curl_setopt ($ch[$i], CURLOPT_HEADER, 1);
            curl_setopt ($ch[$i], CURLOPT_VERBOSE, 0);
            curl_setopt ($ch[$i], CURLOPT_NOBODY, 1);
            curl_setopt ($ch[$i], CURLOPT_CUSTOMREQUEST, 'GET');
            curl_setopt ($ch[$i], CURLOPT_STDERR, $fp);
            curl_setopt ($ch[$i], CURLOPT_FAILONERROR, 0);
            curl_setopt ($ch[$i], CURLOPT_FOLLOWLOCATION, 0);
            curl_setopt ($ch[$i], CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 (via www.askapache.com)');
            curl_setopt ($ch[$i], CURLOPT_INTERFACE, '208.86.158.195');
            curl_setopt ($ch[$i], CURLOPT_HTTPHEADER, $FF_HDR);
            curl_setopt ($ch[$i], CURLOPT_REFERER, 'https://www.askapache.com');
            curl_setopt ($ch[$i], CURLOPT_ENCODING, 0);
            curl_setopt ($ch[$i], CURLOPT_CONNECTTIMEOUT, 45);
            curl_setopt ($ch[$i], CURLOPT_MAXCONNECTS, 5);
            curl_setopt ($ch[$i], CURLOPT_MAXREDIRS, 0);
            curl_multi_add_handle ($mh,$ch[$i]);
        }
        do { ob_start();$r=curl_multi_exec($mh,$active);$t=ob_get_clean();}
		while($r == CURLM_CALL_MULTI_PERFORM || $active);
		if ($r != CURLM_OK) die("Curl multi read error $r");
        foreach ($U as $i => $url) {
			if (curl_errno($ch[$i])) {echo curl_error($ch[$i])."-".curl_errno($ch[$i]);}
			//else $cch=curl_getinfo($ch[$i]);
            curl_multi_remove_handle($mh,$ch[$i]);
            curl_close($ch[$i]);
        }
        curl_multi_close($mh);
        fclose($fp);
		}
	sleep(1);
	return true;
}

php code example 2

This code will download all the files listed in the $urls array to the folder specified by the $saveto variable.

 $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Curl Multi Functions

curl_multi_init

Returns a new cURL multi handle

curl_multi_add_handle

Add a normal cURL handle to a cURL multi handle

curl_multi_exec

Run the sub-connections of the current cURL handle

curl_multi_close

Close a set of cURL handles

curl_multi_getcontent

Return the content of a cURL handle if CURLOPT_RETURNTRANSFER is set

curl_multi_info_read

Get information about the current transfers

curl_multi_select

Get all the sockets associated with the cURL extension, which can then be "selected"

curl_multi_remove_handle

Remove a multi handle from a set of cURL handles