Hello,
I make automatic backups with Google Drive service account using PHP.
For example, manually downloading a 10 MB file to the remote server takes 5 minutes.
It wasn’t this slow before, what could be the reason?
Again, downloading to local computer via PHP is equally slow
Hi Adem,
Can you share some code to review - may help to view the steps being taken for the file downloads.
I shared it on github.
Although the description is in Turkish, it is in the google drive library plugins.
This problem did not exist at first, it was downloading very quickly.
However, with cron jobs, I backup the databases and web directories of 10 websites on Google Drive every night at 00:00.
I am using drive in a Normail gmail account.
Could Google have restricted downloads because I make regular backups every night?
I didn’t make any changes to the codes.
Thanks for be interested
While it downloads a 22 MB file in 10 minutes, the code below downloads it in 1.5 minutes.
function listFiles($secilen_dizin, $service, $folderId, $path = '') {
$resultArray = [];
$results = $service->files->listFiles([
'q' => "'$folderId' in parents",
]);
foreach ($results->getFiles() as $file) {
$filePath = $path . '/' . $file->getName();
if ($file->mimeType == 'application/vnd.google-apps.folder') {
$resultArray = array_merge($resultArray, listFiles($secilen_dizin, $service, $file->getId(), $filePath));
$resultArray[$file->getId()][$file->mimeType] = "/".$secilen_dizin.$filePath;
} else {
$resultArray[$file->getId()][$file->mimeType] = "/".$secilen_dizin.$filePath;
}
}
return $resultArray;
}
if (isset($_POST['yerel_den_secilen_dosya']) && isset($_POST['google_drive_dan_secilen_dosya_id'])) {
$yerel_hedef = rtrim($_POST['yerel_den_secilen_dosya'], '/');
$google_kaynak = ltrim(rtrim($_POST['google_drive_dan_secilen_dosya_adini_goster'],'/'),'/');
$fileId = $_POST['google_drive_dan_secilen_dosya_id'];
if (pathinfo($google_kaynak, PATHINFO_EXTENSION)) {
$file = $service->files->get($fileId, ['fields' => 'id,size']);
if (intval($file->size) <= 0) {
echo "Dosya boyutu geçersiz.";
exit;
}
$fileSize = intval($file->size);
$http = $client->authorize();
$fp = fopen(rtrim($yerel_hedef,'/')."/".$google_kaynak, 'w');
$chunkSizeBytes = 10 * 1024 * 1024; // 10 MB
$chunkStart = 0;
while ($chunkStart < $fileSize) {
$chunkEnd = min($chunkStart + $chunkSizeBytes, $fileSize - 1);
$response = $http->request(
'GET',
sprintf('/drive/v3/files/%s', $fileId),
[
'query' => ['alt' => 'media'],
'headers' => [
'Range' => sprintf('bytes=%s-%s', $chunkStart, $chunkEnd)
]
]
);
$chunkStart = $chunkEnd + 1;
fwrite($fp, $response->getBody()->getContents());
}
fclose($fp);
echo "<br /><b>Yerel </b> ".$yerel_hedef." <b>dizine</b><br />";
echo $google_kaynak." <b>[İNDİRİLDİ]</b>";
} else {
$secilen_dizin = $google_kaynak;
$googleden_secilen_dizin_arr[$fileId]['application/vnd.google-apps.folder'] = "/".$google_kaynak;
$filePathsArray = listFiles($secilen_dizin, $service, $fileId);
$secilen_googleden_secilen_array = array_merge($googleden_secilen_dizin_arr, $filePathsArray);
echo "<br /><b>Yerel </b> ".$yerel_hedef." <b>dizine</b><br />";
foreach ($secilen_googleden_secilen_array as $id => $dosya_tipi_dosya_adi) {
foreach ($dosya_tipi_dosya_adi as $dosya_tipi => $dosya_adi) {
if ($dosya_tipi == 'application/vnd.google-apps.folder') {
if (!file_exists($yerel_hedef.$dosya_adi)) {
mkdir($yerel_hedef.$dosya_adi, 0755, true);
}
}
}
}
foreach ($secilen_googleden_secilen_array as $id => $dosya_tipi_dosya_adi) {
foreach ($dosya_tipi_dosya_adi as $dosya_tipi => $dosya_adi) {
if ($dosya_tipi != 'application/vnd.google-apps.folder') {
$file = $service->files->get($id, ['fields' => 'id,size']);
if (intval($file->size) <= 0) {
echo "Dosya boyutu geçersiz.";
continue;
}
$fileSize = intval($file->size);
$http = $client->authorize();
$fp = fopen($yerel_hedef.$dosya_adi, 'w');
$chunkSizeBytes = 10 * 1024 * 1024;
$chunkStart = 0;
while ($chunkStart < $fileSize) {
$chunkEnd = min($chunkStart + $chunkSizeBytes, $fileSize - 1);
$response = $http->request(
'GET',
sprintf('/drive/v3/files/%s', $id),
[
'query' => ['alt' => 'media'],
'headers' => [
'Range' => sprintf('bytes=%s-%s', $chunkStart, $chunkEnd)
]
]
);
$chunkStart = $chunkEnd + 1;
fwrite($fp, $response->getBody()->getContents());
}
fclose($fp);
echo $dosya_adi." <b>[İNDİRİLDİ]</b><br />";
}
}
}
}
} else {
echo "Kaynak ve indirilecek dizin seçilmelidir";
}
However, I want to use the code below to quickly download larger GB files, but it gives an error. Can you help me?
function listFiles($secilen_dizin, $service, $folderId, $path = '') {
$resultArray = [];
$results = $service->files->listFiles([
'q' => "'$folderId' in parents",
]);
foreach ($results->getFiles() as $file) {
$filePath = $path . '/' . $file->getName();
if ($file->mimeType == 'application/vnd.google-apps.folder') {
$resultArray = array_merge($resultArray, listFiles($secilen_dizin, $service, $file->getId(), $filePath));
$resultArray[$file->getId()][$file->mimeType] = "/".$secilen_dizin.$filePath;
} else {
$resultArray[$file->getId()][$file->mimeType] = "/".$secilen_dizin.$filePath;
}
}
return $resultArray;
}
if (isset($_POST['yerel_den_secilen_dosya']) && isset($_POST['google_drive_dan_secilen_dosya_id'])) {
$yerel_hedef = rtrim($_POST['yerel_den_secilen_dosya'], '/');
$google_kaynak = ltrim(rtrim($_POST['google_drive_dan_secilen_dosya_adini_goster'],'/'),'/');
$fileId = $_POST['google_drive_dan_secilen_dosya_id'];
if (pathinfo($google_kaynak, PATHINFO_EXTENSION)) {
$file = $service->files->get($fileId, ['fields' => 'id,size']);
if (intval($file->size) <= 0) {
echo "Dosya boyutu geçersiz.";
exit;
}
$fileSize = intval($file->size);
$http = $client->authorize();
$fp = fopen(rtrim($yerel_hedef, '/') . "/" . $google_kaynak, 'w');
$chunkSizeBytes = 10 * 1024 * 1024; // 10 MB
$chunkStart = 0;
$multiHandle = curl_multi_init();
while ($chunkStart < $fileSize) {
$chunkEnd = min($chunkStart + $chunkSizeBytes, $fileSize - 1);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, sprintf('https://www.googleapis.com/drive/v3/files/%s?alt=media', $fileId));
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Authorization: Bearer ' . $http->getAccessToken(),
'Range: bytes=' . $chunkStart . '-' . $chunkEnd
]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_WRITEFUNCTION, function ($ch, $data) use ($fp) {
fwrite($fp, $data);
return strlen($data);
});
curl_multi_add_handle($multiHandle, $ch);
$chunkStart = $chunkEnd + 1;
}
// Paralel istekleri gerçekleştirme
do {
$status = curl_multi_exec($multiHandle, $active);
if ($active) {
curl_multi_select($multiHandle);
}
} while ($active && $status == CURLM_OK);
// Temizlik
curl_multi_close($multiHandle);
fclose($fp);
echo "<br /><b>Yerel </b> " . $yerel_hedef . " <b>dizine</b><br />";
echo $google_kaynak . " <b>[İNDİRİLDİ]</b>";
} else {
$secilen_dizin = $google_kaynak;
$googleden_secilen_dizin_arr[$fileId]['application/vnd.google-apps.folder'] = "/" . $google_kaynak;
$filePathsArray = listFiles($secilen_dizin, $service, $fileId);
$secilen_googleden_secilen_array = array_merge($googleden_secilen_dizin_arr, $filePathsArray);
echo "<br /><b>Yerel </b> " . $yerel_hedef . " <b>dizine</b><br />";
foreach ($secilen_googleden_secilen_array as $id => $dosya_tipi_dosya_adi) {
foreach ($dosya_tipi_dosya_adi as $dosya_tipi => $dosya_adi) {
if ($dosya_tipi == 'application/vnd.google-apps.folder') {
if (!file_exists($yerel_hedef . $dosya_adi)) {
mkdir($yerel_hedef . $dosya_adi, 0755, true);
}
}
}
}
foreach ($secilen_googleden_secilen_array as $id => $dosya_tipi_dosya_adi) {
foreach ($dosya_tipi_dosya_adi as $dosya_tipi => $dosya_adi) {
if ($dosya_tipi != 'application/vnd.google-apps.folder') {
$file = $service->files->get($id, ['fields' => 'id,size']);
if (intval($file->size) <= 0) {
echo "Dosya boyutu geçersiz.";
continue;
}
$fileSize = intval($file->size);
$http = $client->authorize();
$fp = fopen($yerel_hedef . $dosya_adi, 'w');
$chunkSizeBytes = 10 * 1024 * 1024;
$chunkStart = 0;
$multiHandle = curl_multi_init();
while ($chunkStart < $fileSize) {
$chunkEnd = min($chunkStart + $chunkSizeBytes, $fileSize - 1);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, sprintf('https://www.googleapis.com/drive/v3/files/%s?alt=media', $id));
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Authorization: Bearer ' . $http->getAccessToken(),
'Range: bytes=' . $chunkStart . '-' . $chunkEnd
]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_WRITEFUNCTION, function ($ch, $data) use ($fp) {
fwrite($fp, $data);
return strlen($data);
});
curl_multi_add_handle($multiHandle, $ch);
$chunkStart = $chunkEnd + 1;
}
// Paralel istekleri gerçekleştirme
do {
$status = curl_multi_exec($multiHandle, $active);
if ($active) {
curl_multi_select($multiHandle);
}
} while ($active && $status == CURLM_OK);
// Temizlik
curl_multi_close($multiHandle);
fclose($fp);
echo $dosya_adi . " <b>[İNDİRİLDİ]</b><br />";
}
}
}
}
} else {
echo "Kaynak ve indirilecek dizin seçilmelidir";
}
I am using Google Drive IP “Service” account and no user confirmation is required
The connection code is located in the file google_drive_setup.pnp on the download page include 'google_drive_setup.php'; I’am adding.
function getClient() {
$client = new \Google\Client();
$client->setApplicationName('Google Drive API PHP Quickstart');
$client->setScopes(\Google\Service\Drive::DRIVE);
$client->setAuthConfig(AUTHCONFIGPATH);
$client->setAccessType('offline');
return $client;
}
$client = getClient();
$service = new \Google\Service\Drive($client);
Thank you for the answer,
Slow download problem is related to encoding.
With the latest coding it is now even faster.
My last code, I want to download very large files with a parallel download system, but I am getting a lot of errors. Can you take a look at the code, where are the mistakes?
Hello,
I think you have to check for high network traffic or server load, which can impede file transfers. Your PHP configuration settings, including max_execution_time, memory_limit, and post_max_size, as inadequate settings may cause timeouts or incomplete downloads. May you consider any potential rate limits or restrictions imposed by the Google Drive API if you’re using it for backups.
These php settings are set to unlimited.
I increased the speed quite a bit, the problem is related to coding.
I’m trying to do parallel downloading for GB size files, but I’m getting an error. The sample code is above.
I would be grateful if you help me.
@adem, the last two replies are generic chatbot generated answers, having nothing to do with your question.
What error are you getting?
PHP Fatal error: Uncaught Error: Call to undefined function GuzzleHttp\Promise\settle() →
Promise\all($promises)->wait();
Promise\settle($promises)->wait();
The above gives error
and I tried both options above
use GuzzleHttp\Client;
use GuzzleHttp\Promise;
class GoogleDriveDownloader {
private $service;
private $client;
private $guzzleClient;
public function __construct($service, $client) {
$this->service = $service;
$this->client = $client;
$this->guzzleClient = new Client();
}
private function listFiles($folderId, $path = '') {
$resultArray = [];
$results = $this->service->files->listFiles([
'q' => "'$folderId' in parents",
'fields' => 'files(id, name, mimeType)'
]);
foreach ($results->getFiles() as $file) {
$filePath = $path . '/' . $file->getName();
if ($file->mimeType == 'application/vnd.google-apps.folder') {
$resultArray = array_merge($resultArray, $this->listFiles($file->getId(), $filePath));
$resultArray[$file->getId()] = ['path' => $filePath, 'mimeType' => $file->mimeType];
} else {
$resultArray[$file->getId()] = ['path' => $filePath, 'mimeType' => $file->mimeType];
}
}
return $resultArray;
}
public function downloadFile($localPath, $googlePath, $fileId) {
$file = $this->service->files->get($fileId, ['fields' => 'id,name,size,mimeType']);
if ($file->mimeType == 'application/vnd.google-apps.folder') {
$filePathsArray = $this->listFiles($fileId, $googlePath);
foreach ($filePathsArray as $id => $fileInfo) {
if ($fileInfo['mimeType'] == 'application/vnd.google-apps.folder') {
$folderPath = rtrim($localPath, '/') . '/' . trim($fileInfo['path'], '/');
if (!file_exists($folderPath)) {
mkdir($folderPath, 0755, true);
}
}
}
foreach ($filePathsArray as $id => $fileInfo) {
if ($fileInfo['mimeType'] != 'application/vnd.google-apps.folder') {
$this->downloadFile($localPath, $fileInfo['path'], $id);
}
}
} else {
if (intval($file->size) <= 0) {
echo "Dosya boyutu geçersiz.";
return;
}
$fileSize = intval($file->size);
$chunkSizeBytes = 10 * 1024 * 1024; // 10 MB
$filePath = rtrim($localPath, '/') . '/' . $googlePath;
$promises = [];
$accessTokenArray = $this->client->getAccessToken();
$accessToken = is_array($accessTokenArray) ? $accessTokenArray['access_token'] : $accessTokenArray;
for ($chunkStart = 0; $chunkStart < $fileSize; $chunkStart += $chunkSizeBytes) {
$chunkEnd = min($chunkStart + $chunkSizeBytes, $fileSize - 1);
$range = sprintf('bytes=%s-%s', $chunkStart, $chunkEnd);
$promises[] = $this->guzzleClient->getAsync(sprintf('https://www.googleapis.com/drive/v3/files/%s?alt=media', $fileId), [
'headers' => [
'Authorization' => 'Bearer ' . $accessToken,
'Range' => $range,
]
])->then(function ($response) use ($filePath, $chunkStart) {
$file = fopen($filePath, 'c');
fseek($file, $chunkStart);
fwrite($file, $response->getBody()->getContents());
fclose($file);
});
}
Promise\all($promises)->wait();
echo "<br /><b>Yerel </b> " . $localPath . " <b>dizine</b><br />";
echo $googlePath . " <b>[İNDİRİLDİ]</b>";
}
}
}
ob_start();
ini_set('memory_limit', '-1');
ignore_user_abort(true);
set_time_limit(3600); // 7200 saniye 120 dakikadır, 3600 1 saat
if (isset($_POST['yerel_den_secilen_dosya']) && isset($_POST['google_drive_dan_secilen_dosya_id'])) {
$yerel_hedef = rtrim($_POST['yerel_den_secilen_dosya'], '/');
$google_kaynak = trim($_POST['google_drive_dan_secilen_dosya_adini_goster'], '/');
$fileId = $_POST['google_drive_dan_secilen_dosya_id'];
$downloader = new GoogleDriveDownloader($service, $client);
$downloader->downloadFile($yerel_hedef, $google_kaynak, $fileId);
} else {
echo "Kaynak ve indirilecek dizin seçilmelidir";
}