Cron job on a large database with PHP, suggestions needed

I've heard about cron job and don't think the actual creation of it will be that hard to make but I've some concerns about how this will work with a large script.

Without going too much off-topic on my project i will stick with the basics about my situation. I need to make a script that every day performs a CURL fetch for data on a remote website and updates an database for each featured member on my website with it. In short, it's approximatively at this time 1000 times the script need to be executed but it will be a larger number as times goes by.

As you can guess, this will take a long time to preform so i'm worried about how the execution will work in a manner of not crashing in the middle of it.

My first thought was to perhaps split the users into groups and make the executions on a small amount of users each time but don't know how this is manageable ( will read on further about the topic when i got some form of confirmation on this).

So, to my question. Do you think there is any way for me to make this happen and do you perhaps have any suggestions 开发者_JAVA百科on how to make this to work efficiently? All help i can get is appreciated. Thank you for your time.

---

bigger cron-jobs with php and mysql needs to be fragmented, since there is no way for you to 'nice' them, (reduce their os priority). Even if you nice the script, the mysql-requests will be executed without this concern.

From what you're describing there's two aspects to consider:

• Congestion of network bandwith
• Congestion of database throughput

I'd recommend a fragmented solution where you call your script from cron more often, and let the script execute only a small amount of the total job. The job should further be canceled (postponed to next run) if i/o-bandwith or cpu-usage is above any limit that may affect response-time to visitors.

regards, /t

---

One Way:

I'm usually against putting logic in the database, but in this case a stored procedure might help. It will run your job faster (since it's a large one) and also you want to lock the tables as you do it. That way, if the script that calls the stored procedure gets hit by cron before the original job was over with it wont edit your database while the first one is running.

---

The actual time can i not give an straight answer on but based on previously experiences this will take longer then the max execution time.

So solve that problem. There's a reason you can have a different php.ini for the command line interface. Then you can simply focus on processing all users in one script.

---

I solved this program using the files of cron job as differents cron jobs with small pieces. If you are using PHP you can set a cron job to domain/cronjob1.php, domain/cronjob2.php limiting the database lets say 10 with

$sql="SELECT * FROM table LIMIT 10"; 

to cronjob 1 and the rest in cronjo2