业务上的需求,我们开发的供应链系统某些业务表也陆续突破了百万级别。
原先使用 \Maatwebsite\Excel 插件导出的效率越来越慢,5w条数据导出基本要达到20min,甚至于30w数据导出基本上都超时。
为了解决这个问题,多种尝试,做出了以下调整:
首先:是为了能导出。
对于百万数据的体量,我还是选择了xlsWriter。
之前在其他文章也分享了laravel的配置方法。
laravel的扩展特别的多,比如larave-excel。
但是很多只是调用了加粗什么的一些比较常用的方法。
像合并单元格等其他比较复杂的有时候调用不了。所以xlswriter的方法选择自己编写。
参考文档:
https://xlswriter-docs.viest.me/zh-cn/nei-cun/gu-ding-nei-cun-mo-shihttps://xlswriter-docs.viest.me/zh-cn/nei-cun/gu-ding-nei-cun-mo-shi
以下贴出我根据PHP-EXCEL文档编写的通用方法
<?php
namespace App\Admin\Actions\Post;
use \Vtiful\Kernel\Excel;
class XlsWriter
{
// 默认宽度
private $defaultWidth = 16;
// 默认高度
private $defaultHeight = 30;
// 默认导出格式
private $exportType = '.xlsx';
// 表头最大层级
private $maxHeight = 1;
// 文件名
private $fileName = null;
// 默认公式行距离数据间隔
private $defaultFormulaTop = 2;
// 数据占用截至行
private $maxDataLine = 2;
// 默认的单元格格式,常规
private $defaultCellFormat = 'general';
// 支持的单元格格式,可扩充
private $allowCellFormat = [
'general' => \PHPExcel_Style_NumberFormat::FORMAT_GENERAL,
'text' => \PHPExcel_Style_NumberFormat::FORMAT_TEXT,
];
// 支持的单元列操作-数据合并
const CELL_ACT_MERGE = 'merge';
// 支持的单元列操作-背景颜色
const CELL_ACT_BACKGROUND = 'background';
// 数据合并开始标识
const ACT_MERGE_START = 'start';
// 数据合并结束标识
const ACT_MERGE_END = 'end';
private $allowCellActs = [
self::CELL_ACT_MERGE,
self::CELL_ACT_BACKGROUND,
];
// 单元格操作集合
private $cellActs = [];
private $xlsObj;
private $fileObject;
private $format;
private $boldIStyle;
private $colManage;
private $lastColumnCode;
public function __construct()
{
// 文件默认输出地址
$path = public_path("download/xlsExcel");
if (!file_exists($path)){
mkdir ($path,0777,true);
}
$config = [
'path' => $path
];
$this->xlsObj = (new \Vtiful\Kernel\Excel($config));
}
/**
* 设置文件名
* @param string $fileName 文件名
* @param string $sheetName 第一个sheet名
* @param bool $memoryMode 是否开启固定内存模式
*/
public function setFileName(string $fileName = '', string $sheetName = 'Sheet1', $memoryMode = false)
{
$fileName = empty($fileName) ? (string)time() : $fileName;
$fileName .= $this->exportType;
$this->fileName = $fileName;
if ($memoryMode){
$this->fileObject = $this->xlsObj->constMemory($fileName, $sheetName);
}else{
$this->fileObject = $this->xlsObj->fileName($fileName, $sheetName);
}
$this->format = (new \Vtiful\Kernel\Format($this->fileObject->getHandle()));
}
/**
* 设置表头
* @param array $header
* @throws \Exception
*/
public function setHeader(array $header)
{
if (empty($header)) {
throw new \Exception('表头数据不能为空');
}
if (is_null($this->fileName)) {
self::setFileName(time());
}
// 获取单元格合并需要的信息
$colManage = self::setHeaderNeedManage($header);
// 完善单元格合并信息
$this->colManage = self::completeColMerge($colManage);
// 设置最后单元格标识
$this->lastColumnCode = self::getColumn(end($this->colManage)['cursorEnd']) . $this->maxHeight;
// 合并单元格
self::queryMergeColumn();
}
/**
* 填充文件数据
* @param array $data
* @throws \Exception
*/
public function setData(array $data)
{
// 起始行
$indexRow = $this->maxHeight + 1;
// 起始列
$indexCol = 0;
foreach ($data as $row => $datum) {
foreach ($datum as $column => $value) {
// 列值为数组,说明有额外操作
if (is_array($value)) {
$val = $value[0];
$act = $value[1];
$pos = self::getColumn($indexCol) . $indexRow;
// 有效行为
// dump($this->allowCellActs);
$availableActs = array_intersect($this->allowCellActs, array_keys($act));
foreach ($availableActs as $availableAct) {
$index = $act['uniqueId'] ?? $indexCol;
switch ($availableAct) {
case self::CELL_ACT_MERGE:
// 数据合并
$this->cellActs[$index][self::CELL_ACT_MERGE][$act[$availableAct]] = $pos;
$this->cellActs[$index][self::CELL_ACT_MERGE]['val'] = $val;
break;
case self::CELL_ACT_BACKGROUND:
// 背景颜色
$this->cellActs[$index][self::CELL_ACT_BACKGROUND][] = [
'row' => $row,
'column' => $column,
'color' => $act[$availableAct],
'val' => $val
];
break;
default:
throw new \Exception('不支持的单元格操作['. $availableAct .']');
}
}
} else {
$this->fileObject->insertText($row + $this->maxHeight, $column, $value);
}
$indexCol++;
}
$indexRow++;
$indexCol = 0;
}
// 执行单元格操作
self::queryCellActs();
$this->maxDataLine = $this->maxHeight + count($data);
}
/**
* 添加Sheet
* @param string $sheetName
*/
public function addSheet(string $sheetName)
{
$this->fileObject->addSheet($sheetName);
}
/**
* 设置公式
* {start}:数据开始行 {end}:数据结束行
* col_title:公式标题所在列标识,从0开始
* title:公式标题
* col_formula:公式结果所在列标识
* formula:公式内容
* @param array $formulas
* @throws \Exception
*/
public function setFormula(array $formulas)
{
if (empty($formulas)) {
throw new \Exception('公式格式错误');
}
$line = $this->maxDataLine + $this->defaultFormulaTop;
foreach ($formulas as $formula) {
if (isset($formula['col_title']) && isset($formula['title'])) {
$this->fileObject->insertText($line, $formula['col_title'], $formula['title']);
}
if (!isset($formula['col_formula']) || !isset($formula['formula']) || empty($formula['formula'])) {
throw new \Exception('公式格式错误');
}
$formula['formula'] = str_ireplace('{start}', $this->maxHeight + 1, $formula['formula']);
$formula['formula'] = str_ireplace('{end}', $this->maxDataLine, $formula['formula']);
$this->fileObject->insertFormula($line, $formula['col_formula'], $formula['formula']);
}
}
/**
* 设置公式行距离数据间隔
* @param $top
*/
public function reBuildFormulaTop(int $top)
{
$this->defaultFormulaTop = $top;
}
/**
* 插入本地图片
* @param int $row
* @param int $column
* @param string $localImagePath
* @param float|int $widthScale
* @param float|int $heightScale
* @throws \Exception
*/
public function setImage(int $row, int $column, string $localImagePath, float $widthScale = 1, float $heightScale = 1)
{
if (!file_exists($localImagePath)) {
throw new \Exception("未检测到图片{$localImagePath}");
}
$this->fileObject->insertImage($row, $column, $localImagePath, $widthScale, $heightScale);
}
/**
* 冻结表头(需放到setHeader后调用)
*/
public function setFreezeHeader()
{
$this->fileObject->freezePanes($this->maxHeight, 0);
}
/**
* 开启过滤选项(需放到setHeader后调用)
*/
public function setFilter($line='A1')
{
$this->fileObject->autoFilter("$line:{$this->lastColumnCode}");
}
/**
* 设置表头加粗(需放到setHeader后调用)
*/
public function setBoldHeader()
{
$this->boldIStyle = $this->format->bold()->toResource();
$this->fileObject->setRow("A1:{$this->lastColumnCode}", $this->defaultHeight, $this->boldIStyle);
}
/**
* 设置表头斜体(需放到setHeader后调用)
*/
public function setItalicHeader()
{
$this->boldIStyle = $this->format->italic()->toResource();
$this->fileObject->setRow("A1:{$this->lastColumnCode}", $this->defaultHeight, $this->boldIStyle);
}
/**
* 设置表头水平居中对齐(需放到setHeader后调用)
*/
public function setAlignCenterHeader()
{
$this->boldIStyle = $this->format->align(\Vtiful\Kernel\Format::FORMAT_ALIGN_CENTER, \Vtiful\Kernel\Format::FORMAT_ALIGN_VERTICAL_CENTER)->toResource();
}
/**
* 设置全局默认样式,需在setData前面调用
*/
public function setDefaultFormatData()
{
$set_data = (new \Vtiful\Kernel\Format($this->fileObject->getHandle()));
$this->fileObject->defaultFormat(
$set_data
->align(\Vtiful\Kernel\Format::FORMAT_ALIGN_CENTER, \Vtiful\Kernel\Format::FORMAT_ALIGN_VERTICAL_CENTER)
->border(\Vtiful\Kernel\Format::BORDER_THIN)
->toResource()
);
}
/**
* 文件密码保护
* @param $password
*/
public function setFileProtection($password = null)
{
$this->fileObject->protection($password);
}
/**
* 保存文件至服务器
*/
public function output()
{
return $this->fileObject->output();
}
/**
* 输出到浏览器
* @param $filePath
* @throws \Exception
*/
public function excelDownload($filePath)
{
$fileName = $this->fileName;
$userBrowser = $_SERVER['HTTP_USER_AGENT'];
if( preg_match('/MSIE/i', $userBrowser)) {
$fileName = urlencode($fileName);
} else {
$fileName = iconv('UTF-8', 'GBK//IGNORE', $fileName);
}
header("Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
header('Content-Disposition: attachment;filename="' . $fileName . '"');
header('Content-Length: ' . filesize($filePath));
header('Content-Transfer-Encoding: binary');
header('Cache-Control: must-revalidate');
header('Cache-Control: max-age=0');
header('Pragma: public');
if (ob_get_contents()) {
ob_clean();
}
flush();
if (copy($filePath, 'php://output') === false) {
throw new \Exception($filePath. '地址出问题了');
}
// 删除本地文件
@unlink($filePath);
exit();
}
/**
* 组装单元格合并需要的信息
* @param $header
* @param int $cursor
* @param int $col
* @param array $colManage
* @param $parentList
* @param $parent
* @throws \Exception
* @return array
*/
private function setHeaderNeedManage($header, $col = 1, &$cursor = 0, &$colManage = [], $parent = null, $parentList = [])
{
foreach ($header as $head) {
if (empty($head['title'])) {
throw new \Exception('表头数据格式有误');
}
if (is_null($parent)) {
// 循环初始化
$parentList = [];
$col = 1;
} else {
// 递归进入,高度和父级集合通过相同父级条件从已有数组中获取,避免递归增加与实际数据不符
foreach ($colManage as $value) {
if ($value['parent'] == $parent) {
$parentList = $value['parentList'];
$col = $value['height'];
break;
}
}
}
// 单元格标识
$column = $this->getColumn($cursor) . $col;
// 单元格格式
$format = $this->allowCellFormat[$this->defaultCellFormat];
if (!empty($head['format'])) {
if (!isset($this->allowCellFormat[$head['format']])) {
throw new \Exception("不支持的单元格格式{$head['format']}");
}
$format = $this->allowCellFormat[$head['format']];
}
// 组装单元格需要的各种信息
$colManage[$column] = [
'title' => $head['title'], // 标题
'cursor' => $cursor, // 游标
'cursorEnd' => $cursor, // 结束游标
'height' => $col, // 高度
'width' => $this->defaultWidth, // 宽度
'format' => $format, // 单元格格式
'mergeStart' => $column, // 合并开始标识
'hMergeEnd' => $column, // 横向合并结束标识
'zMergeEnd' => $column, // 纵向合并结束标识
'parent' => $parent, // 父级标识
'parentList' => $parentList, // 父级集合
];
if (!empty($head['children']) && is_array($head['children'])) {
// 有下级,高度加一
$col += 1;
// 当前标识加入父级集合
$parentList[] = $column;
$this->setHeaderNeedManage($head['children'], $col, $cursor,$colManage, $column, $parentList);
} else {
// 没有下级,游标加一
$cursor += 1;
}
}
return $colManage;
}
/**
* 完善单元格合并信息
* @param $colManage
* @return array
*/
private function completeColMerge($colManage)
{
$this->maxHeight = max(array_column($colManage, 'height'));
$parentManage = array_column($colManage, 'parent');
foreach ($colManage as $index => $value) {
// 设置横向合并结束范围:存在父级集合,把所有父级的横向合并结束范围设置为当前单元格
if (!is_null($value['parent']) && !empty($value['parentList'])) {
foreach ($value['parentList'] as $parent) {
$colManage[$parent]['hMergeEnd'] = self::getColumn($value['cursor']) . $colManage[$parent]['height'];
$colManage[$parent]['cursorEnd'] = $value['cursor'];
}
}
// 设置纵向合并结束范围:当前高度小于最大高度 且 不存在以当前单元格标识作为父级的项
$checkChildren = array_search($index, $parentManage);
if ($value['height'] < $this->maxHeight && !$checkChildren) {
$colManage[$index]['zMergeEnd'] = self::getColumn($value['cursor']) . $this->maxHeight;
}
}
return $colManage;
}
/**
* 合并单元格
*/
private function queryMergeColumn()
{
foreach ($this->colManage as $value) {
$this->fileObject->mergeCells("{$value['mergeStart']}:{$value['zMergeEnd']}", $value['title']);
$this->fileObject->mergeCells("{$value['mergeStart']}:{$value['hMergeEnd']}", $value['title']);
// 设置单元格需要的宽度
if ($value['cursor'] != $value['cursorEnd']) {
$value['width'] = ($value['cursorEnd'] - $value['cursor'] + 1) * $this->defaultWidth;
}
// 设置单元格格式
$formatCell = (new \Vtiful\Kernel\Format($this->fileObject->getHandle()));
$boldStyle = $formatCell->number($value['format'])->toResource();
// 设置列单元格样式
$toColumnStart = self::getColumn($value['cursor']);
$toColumnEnd = self::getColumn($value['cursorEnd']);
$this->fileObject->setColumn("{$toColumnStart}:{$toColumnEnd}", $value['width'], $boldStyle);
}
}
/**
* 执行单元格操作
*/
private function queryCellActs()
{
if (!empty($this->cellActs)) {
foreach ($this->cellActs as $actNote) {
// dd($this->cellActs);
$tmpActStyle = (new \Vtiful\Kernel\Format($this->fileObject->getHandle()));
// 背景颜色
if (isset($actNote[self::CELL_ACT_BACKGROUND])) {
foreach ($actNote[self::CELL_ACT_BACKGROUND] as $item) {
// 支持颜色常量
$tmpActStyle->background($this->backgroundConst($item['color']))
->border(\Vtiful\Kernel\Format::BORDER_THIN);
$this->fileObject->insertText($item['row'] + $this->maxHeight, $item['column'], $item['val'], '', $tmpActStyle->toResource());
}
}
// 数据合并
if (isset($actNote[self::CELL_ACT_MERGE])) {
if (!empty($actNote[self::CELL_ACT_MERGE][self::ACT_MERGE_START]) && !empty($actNote[self::CELL_ACT_MERGE][self::ACT_MERGE_END])) {
// 合并样式:水平左对齐,垂直居中对齐
$tmpActStyle->align(\Vtiful\Kernel\Format::FORMAT_ALIGN_CENTER, \Vtiful\Kernel\Format::FORMAT_ALIGN_VERTICAL_CENTER)
->border(\Vtiful\Kernel\Format::BORDER_THIN);
$this->fileObject->mergeCells(
"{$actNote[self::CELL_ACT_MERGE][self::ACT_MERGE_START]}:{$actNote[self::CELL_ACT_MERGE][self::ACT_MERGE_END]}",
$actNote[self::CELL_ACT_MERGE]['val'],
$tmpActStyle->toResource()
);
}
}
}
$this->cellActs = [];
}
}
/**
* 颜色常量转换
* @param $color
* @return mixed
*/
private function backgroundConst($color)
{
$const = [
'black' => \Vtiful\Kernel\Format::COLOR_BLACK, // 黑色
'blue' => \Vtiful\Kernel\Format::COLOR_BLUE, // 蓝色
'brown' => \Vtiful\Kernel\Format::COLOR_BROWN, // 棕色
'cyan' => \Vtiful\Kernel\Format::COLOR_CYAN, // 青色
'gray' => \Vtiful\Kernel\Format::COLOR_GRAY, // 灰色
'green' => \Vtiful\Kernel\Format::COLOR_GREEN, // 绿色
'lime' => \Vtiful\Kernel\Format::COLOR_LIME, // 石灰
'magenta' => \Vtiful\Kernel\Format::COLOR_MAGENTA, // 洋红
'navy' => \Vtiful\Kernel\Format::COLOR_NAVY, // 深蓝
'orange' => \Vtiful\Kernel\Format::COLOR_ORANGE, // 橙色
'pink' => \Vtiful\Kernel\Format::COLOR_PINK, // 粉红
'purple' => \Vtiful\Kernel\Format::COLOR_PURPLE, // 紫色
'red' => \Vtiful\Kernel\Format::COLOR_RED, // 红色
'silver' => \Vtiful\Kernel\Format::COLOR_SILVER, // 银色
'white' => \Vtiful\Kernel\Format::COLOR_WHITE, // 白色
'yellow' => \Vtiful\Kernel\Format::COLOR_YELLOW, // 黄色
];
return $const[$color] ?? $color;
}
/**
* 获取单元格列标识
* @param $num
* @return string
*/
private function getColumn($num)
{
return Excel::stringFromColumnIndex($num);
}
}
调用方法:
<?php
$detail_header = [[
'title' => '应收账明细',
'children' => [
['title' => "项目"],
['title' => "日期"],
['title' => "摘要"],
['title' => "应收金额"],
['title' => "是否已开票"],
['title' => "已收金额"],
['title' => "未收金额"],
['title' => "备注"]
]
]];
if (!empty($total_res)){
foreach ($total_res as $key => $val) {
$data_total_pre[$val['store_id']][$val['account_title_id']][] = array(
'account_receivable_id' =>$val['id'] ?? '',
'store_id' =>$val['store_id'] ?? '',
'storeName' =>$val['store']['name'] ?? '',
'accountTitle' =>$val['account_title']['title'] ?? '',
'month' =>$val['month'] ?? '',
'amount' =>$val['amount'] ?? '',
'wait_amount_sum' =>$val['wait_amount_sum'] ?? '',
'actually_amount' =>$val['actually_amount'] ?? '',
'account_title_id' =>$val['account_title_id'] ?? '',
);
}
if (!empty($data_total_pre)){
foreach ($data_total_pre as $store_total_key => $store_total_value){
$store_prev = true;
$end_title_id = $store_total_value;
$end_title_id = end($end_title_id)[0]['account_title_id'];
$store_amount = 0;
$store_wait_amount_sum = 0;
$store_actually_amount = 0;
foreach ($store_total_value as $title_total_key => $title_total_value){
$end_receivable_id = $title_total_value;
$end_receivable_id = end($end_receivable_id)['account_receivable_id'];
foreach ($title_total_value as $total_key => $total_value){
$temp_total = array(
$total_value['storeName'],
$total_value['accountTitle'],
$total_value['month'],
$total_value['amount'],
$total_value['wait_amount_sum'],
$total_value['actually_amount'],
);
$store_next = false;
if ($total_value['account_title_id'] == $end_title_id && $total_value['account_receivable_id'] == $end_receivable_id){
$store_next = true;
}
if ($store_prev !== $store_next){
if ($store_prev) $temp_total[0] = [$temp_total[0],['merge' => 'start','uniqueId' => $total_value['store_id']]];
if ($store_next) $temp_total[0] = [$temp_total[0],['merge' => 'end','uniqueId' => $total_value['store_id']]];
}
if (count($title_total_value) != 1){
if ($total_key == 0) $temp_total[1] = [$temp_total[1],['merge' => 'start','uniqueId' => $total_value['store_id'] . $total_value['account_title_id']]];
if ($total_key == count($title_total_value) - 1) $temp_total[1] = [$temp_total[1],['merge' => 'end','uniqueId' => $total_value['store_id'] . $total_value['account_title_id']]];
}
$store_amount += $total_value['amount'];
$store_wait_amount_sum += $total_value['wait_amount_sum'];
$store_actually_amount += $total_value['actually_amount'];
$data_total[] = $temp_total;
$store_prev = false;
}
}
//插入小计
$data_total[] = array(
['小计' , ['merge' => 'start','uniqueId' => $store_total_key . 'subtotal']],
'',
['小计' , ['merge' => 'end','uniqueId' => $store_total_key . 'subtotal','background' => 0xBFBFBF]],
[round($store_amount, 2), ['uniqueId' => $store_total_key . 'subtotal','background' => 0xBFBFBF]],
[round($store_wait_amount_sum, 2), ['uniqueId' => $store_total_key . 'subtotal','background' => 0xBFBFBF]],
[round($store_actually_amount, 2), ['uniqueId' => $store_total_key . 'subtotal', 'background' => 0xBFBFBF]],
);
$sum_amount += $store_amount;
$sum_wait_amount_sum += $store_wait_amount_sum;
$sum_actually_amount += $store_actually_amount;
}
}
}
try {
$fileName = "应收账表格导出" . date("YmdHis");
$xlsWriterServer = new XlsWriter();
$xlsWriterServer->setFileName($fileName, '应收账汇总');
$xlsWriterServer->setHeader($total_header); //这里可以使用新的header
$xlsWriterServer->setFreezeHeader(); // 冻结表头
$xlsWriterServer->setDefaultFormatData();
$xlsWriterServer->setData($data_total); // 这里也可以根据新的header定义数据格式
$xlsWriterServer->setBoldHeader(); // 设置表头加粗
$xlsWriterServer->setFilter('A2'); // 表头开启过滤选项
$xlsWriterServer->setAlignCenterHeader(); // 设置表头水平居中
// $xlsWriterServer->setItalicHeader(); // 设置表头斜体
// $xlsWriterServer->setFileProtection('testpwd'); // 设置文件解除锁定保护密码
$xlsWriterServer->addSheet('应收账明细');
$xlsWriterServer->setHeader($detail_header);
$xlsWriterServer->setFilter('A2'); // 表头开启过滤选项
$xlsWriterServer->setFreezeHeader(); // 冻结表头
$xlsWriterServer->setAlignCenterHeader(); // 设置表头水平居中
$xlsWriterServer->setData($detail_column);
$xlsWriterServer->setBoldHeader(); // 设置表头加粗
$xlsWriterServer->setFormula($formulas_detail); // 设置公式
$xlsWriterServer->reBuildFormulaTop(3); // 设置公式行距离数据行的间隔(默认2),这里使第二个公式数组在第一个公式下面
// $xlsWriterServer->setFormula($formulas_test_two);
$filePath = $xlsWriterServer->output(); // 保存到服务器
$xlsWriterServer->excelDownload($filePath); // 输出到浏览器
} catch (\Exception $e) {
dd($e->getTrace());
exit($e->getMessage());
}
合并方法:
['merge' => 'start','background' => 0xBFBFBF,'uniqueId' => $total_value['store_id']]];
使用php-xlswriter后,基本上导出百万级别excel是50秒左右,导出5w条速度1秒多。
速度提升之后,发现,如果数据量小的话还好,但是如果一旦达到了百万级别。虽然可以导出,但是对服务器消耗特别大,基本上消耗了8G内存左右。
所以继续思考:
2:减少内存消耗
第一步:
DB::connection()->disableQueryLog();
意思是Laravel框架默认存储每次请求(每次命令行执行也相当于一次请求)的所有数据库查询语句!!!
在普通的http中数据库请求语句并不多,所有不会导致问题,但是需要大量数据库查询的命令行工具就显然不能这么干,解决方法是禁用query日志:
DB::connection()->disableQueryLog(); //禁用query log
第二步:
foreach遍历由传统改为游标
$detail_res->cursor()->toArray()
cursor 的原理
cursor 的实现使用了 yield 关键字,yield 关键字是生成器函数的核心,它的调用形式跟 return 很像,不同之处在于 return 会返回值并且终止函数执行,而 yield 会返回值给循环调用生成器的代码并且只是暂停生成器函数.
cursor() 的代码如下
/**
* Get a generator for the given query.
*
* @return \Generator
*/
public function cursor()
{
foreach ($this->applyScopes()->query->cursor() as $record) {
yield $this->newModelInstance()->newFromBuilder($record);
}
}
由于使用了 yield
关键字,在循环 cursor
生成器的时候,可以渐进式的处理数据,即使在内存很小的情况下,也可以轻松处理千万级的数据!真的是非常方便哦!
第三部:
导出使用xlswriter的固定内存模式
$xlsWriterServer->setFileName($fileName, '采购数据统计', true);
通过一波操作,导出时间虽然增加到了1min5s,满了十几秒,但是内存占用最多在2G多,也还可以接受!
第四步:
减少全盘导出的机会,跟相关使用人员沟通之后,把时间跨度控制在365天内,超过会提示。让他们最多只是导出一年的数据,以防以后达到千万级别会消耗更大的内存。
第五步:
这一步我保留还没使用,就是异步导出 + chunk分块。
结束!